Как я могу нормализовать символ EOL в Java?

Question

У меня есть сервер Linux и много клиентов со многими операционными системами. Сервер принимает входной файл от клиентов. У Linux есть конец строки с символом LF, а у Mac - конец строки с символом CR, и В Windows есть конец строки с символом CR + LF

Сервер нуждается в качестве конца строки в LF. Используя Java, я хочу убедиться, что файл всегда будет использовать linux eol char LF. Как мне этого добиться?

Answer 1

Объединение двух ответов (по Visage & eumiro):

РЕДАКТИРОВАТЬ: После прочтения комментария. линия. System.getProperty("line.separator") тогда бесполезен.
Перед отправкой файла на сервер откройте его, замените все EOL и выполните обратную запись
Обязательно используйте DataStreams и пишите в двоичном формате

String fileString;
//..
//read from the file
//..
//for windows
fileString = fileString.replaceAll("\\r\\n", "\n");
fileString = fileString.replaceAll("\\r", "\n");
//..
//write to file in binary mode.. something like:
DataOutputStream os = new DataOutputStream(new FileOutputStream("fname.txt"));
os.write(fileString.getBytes());
//..
//send file
//..

Метод replaceAll имеет два аргумента: первый - строка для замены, а второй - замена. Но первое рассматривается как регулярное выражение, поэтому '\' интерпретируется таким образом. Итак:

"\\r\\n" is converted to "\r\n" by Regex
"\r\n" is converted to CR+LF by Java

Answer 2

Не могли бы вы попробовать это?

content.replaceAll("\\r\\n?", "\n")

Answer 3

Пришлось сделать это для недавнего проекта.Приведенный ниже метод нормализует окончания строк в данном файле до окончания строк, указанного в ОС, на которой работает JVM.Поэтому, если у вас JVM работает в Linux, все строки будут нормализованы до LF (\ n).

Также работает с очень большими файлами из-за использования буферизованных потоков.

public static void normalizeFile(File f) {      
    File temp = null;
    BufferedReader bufferIn = null;
    BufferedWriter bufferOut = null;        

    try {           
        if(f.exists()) {
            // Create a new temp file to write to
            temp = new File(f.getAbsolutePath() + ".normalized");
            temp.createNewFile();

            // Get a stream to read from the file un-normalized file
            FileInputStream fileIn = new FileInputStream(f);
            DataInputStream dataIn = new DataInputStream(fileIn);
            bufferIn = new BufferedReader(new InputStreamReader(dataIn));

            // Get a stream to write to the normalized file
            FileOutputStream fileOut = new FileOutputStream(temp);
            DataOutputStream dataOut = new DataOutputStream(fileOut);
            bufferOut = new BufferedWriter(new OutputStreamWriter(dataOut));

            // For each line in the un-normalized file
            String line;
            while ((line = bufferIn.readLine()) != null) {
                // Write the original line plus the operating-system dependent newline
                bufferOut.write(line);
                bufferOut.newLine();                                
            }

            bufferIn.close();
            bufferOut.close();

            // Remove the original file
            f.delete();

            // And rename the original file to the new one
            temp.renameTo(f);
        } else {
            // If the file doesn't exist...
            log.warn("Could not find file to open: " + f.getAbsolutePath());
        }
    } catch (Exception e) {
        log.warn(e.getMessage(), e);
    } finally {
        // Clean up, temp should never exist
        FileUtils.deleteQuietly(temp);
        IOUtils.closeQuietly(bufferIn);
        IOUtils.closeQuietly(bufferOut);
    }
}

Answer 4

Вот всеобъемлющий вспомогательный класс для решения проблем EOL.Это частично основано на решении, опубликованном tyjen.

import java.io.BufferedInputStream;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;

import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;

/**
 * Helper class to deal with end-of-line markers in text files.
 * 
 * Loosely based on these examples:
 *  - http://stackoverflow.com/a/9456947/1084488 (cc by-sa 3.0)
 *  - http://svn.apache.org/repos/asf/tomcat/trunk/java/org/apache/tomcat/buildutil/CheckEol.java (Apache License v2.0)
 * 
 * This file is posted here to meet the "ShareAlike" requirement of cc by-sa 3.0:
 *    http://stackoverflow.com/a/27930311/1084488
 * 
 * @author Matthias Stevens
 */
public class EOLUtils
{

    /**
     * Unix-style end-of-line marker (LF)
     */
    private static final String EOL_UNIX = "\n";

    /**
     * Windows-style end-of-line marker (CRLF)
     */
    private static final String EOL_WINDOWS = "\r\n";

    /**
     * "Old Mac"-style end-of-line marker (CR)
     */
    private static final String EOL_OLD_MAC = "\r";

    /**
     * Default end-of-line marker on current system
     */
    private static final String EOL_SYSTEM_DEFAULT = System.getProperty( "line.separator" );

    /**
     * The support end-of-line marker modes
     */
    public static enum Mode
    {
        /**
         * Unix-style end-of-line marker ("\n")
         */
        LF,

        /**
         * Windows-style end-of-line marker ("\r\n") 
         */
        CRLF,

        /**
         * "Old Mac"-style end-of-line marker ("\r")
         */
        CR
    }

    /**
     * The default end-of-line marker mode for the current system
     */
    public static final Mode SYSTEM_DEFAULT = ( EOL_SYSTEM_DEFAULT.equals( EOL_UNIX ) ? Mode.LF : ( EOL_SYSTEM_DEFAULT
        .equals( EOL_WINDOWS ) ? Mode.CRLF : ( EOL_SYSTEM_DEFAULT.equals( EOL_OLD_MAC ) ? Mode.CR : null ) ) );
    static
    {
        // Just in case...
        if ( SYSTEM_DEFAULT == null )
        {
            throw new IllegalStateException( "Could not determine system default end-of-line marker" );
        }
    }

    /**
     * Determines the end-of-line {@link Mode} of a text file.
     * 
     * @param textFile the file to investigate
     * @return the end-of-line {@link Mode} of the given file, or {@code null} if it could not be determined
     * @throws Exception
     */
    public static Mode determineEOL( File textFile )
        throws Exception
    {
        if ( !textFile.exists() )
        {
            throw new IOException( "Could not find file to open: " + textFile.getAbsolutePath() );
        }

        FileInputStream fileIn = new FileInputStream( textFile );
        BufferedInputStream bufferIn = new BufferedInputStream( fileIn );
        try
        {
            int prev = -1;
            int ch;
            while ( ( ch = bufferIn.read() ) != -1 )
            {
                if ( ch == '\n' )
                {
                    if ( prev == '\r' )
                    {
                        return Mode.CRLF;
                    }
                    else
                    {
                        return Mode.LF;
                    }
                }
                else if ( prev == '\r' )
                {
                    return Mode.CR;
                }
                prev = ch;
            }
            throw new Exception( "Could not determine end-of-line marker mode" );
        }
        catch ( IOException ioe )
        {
            throw new Exception( "Could not determine end-of-line marker mode", ioe );
        }
        finally
        {
            // Clean up:
            IOUtils.closeQuietly( bufferIn );
        }
    }

    /**
     * Checks whether the given text file has Windows-style (CRLF) line endings.
     * 
     * @param textFile the file to investigate
     * @return
     * @throws Exception
     */
    public static boolean hasWindowsEOL( File textFile )
        throws Exception
    {
        return Mode.CRLF.equals( determineEOL( textFile ) );
    }

    /**
     * Checks whether the given text file has Unix-style (LF) line endings.
     * 
     * @param textFile the file to investigate
     * @return
     * @throws Exception
     */
    public static boolean hasUnixEOL( File textFile )
        throws Exception
    {
        return Mode.LF.equals( determineEOL( textFile ) );
    }

    /**
     * Checks whether the given text file has "Old Mac"-style (CR) line endings.
     * 
     * @param textFile the file to investigate
     * @return
     * @throws Exception
     */
    public static boolean hasOldMacEOL( File textFile )
        throws Exception
    {
        return Mode.CR.equals( determineEOL( textFile ) );
    }

    /**
     * Checks whether the given text file has line endings that conform to the system default mode (e.g. LF on Unix).
     * 
     * @param textFile the file to investigate
     * @return
     * @throws Exception
     */
    public static boolean hasSystemDefaultEOL( File textFile )
        throws Exception
    {
        return SYSTEM_DEFAULT.equals( determineEOL( textFile ) );
    }

    /**
     * Convert the line endings in the given file to Unix-style (LF).
     * 
     * @param textFile the file to process
     * @throws IOException
     */
    public static void convertToUnixEOL( File textFile )
        throws IOException
    {
        convertLineEndings( textFile, EOL_UNIX );
    }

    /**
     * Convert the line endings in the given file to Windows-style (CRLF).
     * 
     * @param textFile the file to process
     * @throws IOException
     */
    public static void convertToWindowsEOL( File textFile )
        throws IOException
    {
        convertLineEndings( textFile, EOL_WINDOWS );
    }

    /**
     * Convert the line endings in the given file to "Old Mac"-style (CR).
     * 
     * @param textFile the file to process
     * @throws IOException
     */
    public static void convertToOldMacEOL( File textFile )
        throws IOException
    {
        convertLineEndings( textFile, EOL_OLD_MAC );
    }

    /**
     * Convert the line endings in the given file to the system default mode.
     * 
     * @param textFile the file to process
     * @throws IOException
     */
    public static void convertToSystemEOL( File textFile )
        throws IOException
    {
        convertLineEndings( textFile, EOL_SYSTEM_DEFAULT );
    }

    /**
     * Line endings conversion method.
     * 
     * @param textFile the file to process
     * @param eol the end-of-line marker to use (as a {@link String})
     * @throws IOException 
     */
    private static void convertLineEndings( File textFile, String eol )
        throws IOException
    {
        File temp = null;
        BufferedReader bufferIn = null;
        BufferedWriter bufferOut = null;

        try
        {
            if ( textFile.exists() )
            {
                // Create a new temp file to write to
                temp = new File( textFile.getAbsolutePath() + ".normalized" );
                temp.createNewFile();

                // Get a stream to read from the file un-normalized file
                FileInputStream fileIn = new FileInputStream( textFile );
                DataInputStream dataIn = new DataInputStream( fileIn );
                bufferIn = new BufferedReader( new InputStreamReader( dataIn ) );

                // Get a stream to write to the normalized file
                FileOutputStream fileOut = new FileOutputStream( temp );
                DataOutputStream dataOut = new DataOutputStream( fileOut );
                bufferOut = new BufferedWriter( new OutputStreamWriter( dataOut ) );

                // For each line in the un-normalized file
                String line;
                while ( ( line = bufferIn.readLine() ) != null )
                {
                    // Write the original line plus the operating-system dependent newline
                    bufferOut.write( line );
                    bufferOut.write( eol ); // write EOL marker
                }

                // Close buffered reader & writer:
                bufferIn.close();
                bufferOut.close();

                // Remove the original file
                textFile.delete();

                // And rename the original file to the new one
                temp.renameTo( textFile );
            }
            else
            {
                // If the file doesn't exist...
                throw new IOException( "Could not find file to open: " + textFile.getAbsolutePath() );
            }
        }
        finally
        {
            // Clean up, temp should never exist
            FileUtils.deleteQuietly( temp );
            IOUtils.closeQuietly( bufferIn );
            IOUtils.closeQuietly( bufferOut );
        }
    }

}

Answer 5

Используйте

System.getProperty("line.separator")

Это даст вам (локальный) символ (ы) EOL. Затем вы можете использовать анализ несовершенного текста, чтобы определить, какой это «вкус», и соответственно преобразовать его.

Либо заставьте своих клиентов стандартизироваться!

Answer 6

Хотя String.replaceAll () проще для кода, он должен работать лучше, поскольку не проходит через инфраструктуру регулярных выражений.

    /**
 * Accepts a non-null string and returns the string with all end-of-lines
 * normalized to a \n.  This means \r\n and \r will both be normalized to \n.
 * <p>
 *     Impl Notes:  Although regex would have been easier to code, this approach
 *     will be more efficient since it's purpose built for this use case.  Note we only
 *     construct a new StringBuilder and start appending to it if there are new end-of-lines
 *     to be normalized found in the string.  If there are no end-of-lines to be replaced
 *     found in the string, this will simply return the input value.
 * </p>
 *
 * @param inputValue !null, input value that may or may not contain new lines
 * @return the input value that has new lines normalized
 */
static String normalizeNewLines(String inputValue){
    StringBuilder stringBuilder = null;
    int index = 0;
    int len = inputValue.length();
    while (index < len){
        char c = inputValue.charAt(index);
        if (c == '\r'){
            if (stringBuilder == null){
                stringBuilder = new StringBuilder();
                // build up the string builder so it contains all the prior characters
                stringBuilder.append(inputValue.substring(0, index));
            }
            if ((index + 1 < len) &&
                inputValue.charAt(index + 1) == '\n'){
                // this means we encountered a \r\n  ... move index forward one more character
                index++;
            }
            stringBuilder.append('\n');
        }else{
            if (stringBuilder != null){
                stringBuilder.append(c);
            }
        }
        index++;
    }
    return stringBuilder == null ? inputValue : stringBuilder.toString();
}

Answer 7

решение для изменения файла, заканчивающегося рекурсивным поиском по пути

package handleFileLineEnd;

import java.io.File;
import java.io.IOException;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.OpenOption;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List;

import sun.awt.image.BytePackedRaster;

public class handleFileEndingMain {

    static int carriageReturnTotal;
    static int newLineTotal;

    public static void main(String[] args)  throws IOException
    {       
        processPath("c:/temp/directories");

        System.out.println("carriageReturnTotal  (files have issue): " + carriageReturnTotal);

        System.out.println("newLineTotal: " + newLineTotal);
    }

    private static void processPath(String path) throws IOException
    {
        File dir = new File(path);
        File[] directoryListing = dir.listFiles();

        if (directoryListing != null) {
            for (File child : directoryListing) {
                if (child.isDirectory())                
                    processPath(child.toString());              
                else
                    checkFile(child.toString());
            }
        } 


    }

    private static void checkFile(String fileName) throws IOException
    {
        Path path = FileSystems.getDefault().getPath(fileName);

        byte[] bytes= Files.readAllBytes(path);

        for (int counter=0; counter<bytes.length; counter++)
        {
            if (bytes[counter] == 13)
            {
                carriageReturnTotal = carriageReturnTotal + 1;

                System.out.println(fileName);
                modifyFile(fileName);
                break;
            }
            if (bytes[counter] == 10)
            {
                newLineTotal = newLineTotal+ 1;
                //System.out.println(fileName);
                break;
            }
        }

    }

    private static void modifyFile(String fileName) throws IOException
    {

        Path path = Paths.get(fileName);
        Charset charset = StandardCharsets.UTF_8;

        String content = new String(Files.readAllBytes(path), charset);
        content = content.replaceAll("\r\n", "\n");
        content = content.replaceAll("\r", "\n");
        Files.write(path, content.getBytes(charset));
    }
}

Answer 8

public static String normalize(String val) {
    return val.replace("\r\n", "\n")
            .replace("\r", "\n");
}

Для HTML:

public static String normalize(String val) {
    return val.replace("\r\n", "<br/>")
            .replace("\n", "<br/>")
            .replace("\r", "<br/>");
}