Newline

From Wikipedia for FEVERv2
Jump to navigation Jump to search

For other uses, see New Line (disambiguation). Newline_sentence_0

Newline (frequently called line ending, end of line (EOL), line feed, or line break) is a control character or sequence of control characters in a character encoding specification (e.g. ASCII or EBCDIC) that is used to signify the end of a and the start of a new one. Newline_sentence_1

Some text editors set this special character when pressing the ↵ Enter key. Newline_sentence_2

When displaying (or printing) a , this control character causes the text editor to show the following characters in a new line. Newline_sentence_3

History Newline_section_0

In the mid-1800s, long before the advent of teleprinters and teletype machines, Morse code operators or telegraphists invented and used Morse code prosigns to encode white space text formatting in formal written text messages. Newline_sentence_4

In particular the Morse prosign represented by the concatenation of two literal textual Morse code "A" characters sent without the normal inter-character spacing is used in Morse code to encode and indicate a new line in a formal text message. Newline_sentence_5

Later in the age of modern teleprinters standardized character set control codes were developed to aid in white space text formatting. Newline_sentence_6

ASCII was developed simultaneously by the International Organization for Standardization (ISO) and the American Standards Association (ASA), the latter being the predecessor organization to American National Standards Institute (ANSI). Newline_sentence_7

During the period of 1963 to 1968, the ISO draft standards supported the use of either CR+LF or LF alone as a newline, while the ASA drafts supported only CR+LF. Newline_sentence_8

The sequence CR+LF was commonly used on many early computer systems that had adopted Teletype machines—typically a Teletype Model 33 ASR—as a console device, because this sequence was required to position those printers at the start of a new line. Newline_sentence_9

The separation of newline into two functions concealed the fact that the print head could not return from the far right to the beginning of the next line in time to print the next character. Newline_sentence_10

Any character printed after a CR would often print as a smudge in the middle of the page while the print head was still moving the carriage back to the first position. Newline_sentence_11

"The solution was to make the newline two characters: CR to move the carriage to column one, and LF to move the paper up." Newline_sentence_12

In fact, it was often necessary to send extra characters—extraneous CRs or NULs—which are ignored but give the print head time to move to the left margin. Newline_sentence_13

Many early video displays also required multiple character times to scroll the display. Newline_sentence_14

On such systems, applications had to talk directly to the Teletype machine and follow its conventions since the concept of device drivers hiding such hardware details from the application was not yet well developed. Newline_sentence_15

Therefore, text was routinely composed to satisfy the needs of Teletype machines. Newline_sentence_16

Most minicomputer systems from DEC used this convention. Newline_sentence_17

CP/M also used it in order to print on the same terminals that minicomputers used. Newline_sentence_18

From there MS-DOS (1981) adopted CP/M's CR+LF in order to be compatible, and this convention was inherited by Microsoft's later Windows operating system. Newline_sentence_19

The Multics operating system began development in 1964 and used LF alone as its newline. Newline_sentence_20

Multics used a device driver to translate this character to whatever sequence a printer needed (including extra padding characters), and the single byte was more convenient for programming. Newline_sentence_21

What seems like a more obvious choice—CR—was not used, as CR provided the useful function of overprinting one line with another to create boldface and strikethrough effects. Newline_sentence_22

Perhaps more importantly, the use of LF alone as a line terminator had already been incorporated into drafts of the eventual ISO/IEC 646 standard. Newline_sentence_23

Unix followed the Multics practice, and later Unix-like systems followed Unix. Newline_sentence_24

This created conflicts between Windows and Unix-like OSes, whereby files composed on one OS can't be properly formatted or interpreted by another OS (for example a UNIX shell script written in a Windows text editor like Notepad). Newline_sentence_25

Representation Newline_section_1

The concepts of carriage return (CR) and line feed (LF) are closely associated and can be considered either separately or together. Newline_sentence_26

In the physical media of typewriters and printers, two axes of motion, "down" and "across", are needed to create a new line on the page. Newline_sentence_27

Although the design of a machine (typewriter or printer) must consider them separately, the abstract logic of software can combine them together as one event. Newline_sentence_28

This is why a newline in character encoding can be defined as CR and LF combined into one (commonly called CR+LF or CRLF). Newline_sentence_29

Some character sets provide a separate newline character code. Newline_sentence_30

EBCDIC, for example, provides an NL character code in addition to the CR and LF codes. Newline_sentence_31

Unicode, in addition to providing the ASCII CR and LF control codes, also provides a "next line" (NEL) control code, as well as control codes for "line separator" and "paragraph separator" markers. Newline_sentence_32

Newline_table_general_0

Software applications and operating system representation of a newline with one or two control charactersNewline_table_caption_0
Operating systemNewline_header_cell_0_0_0 Character encodingNewline_header_cell_0_0_1 AbbreviationNewline_header_cell_0_0_2 hex valueNewline_header_cell_0_0_3 dec valueNewline_header_cell_0_0_4 Escape sequenceNewline_header_cell_0_0_5
Unix and Unix-like systems (Linux, macOS, FreeBSD, AIX, Xenix, etc.), Multics, BeOS, Amiga, RISC OS, and othersNewline_header_cell_0_1_0 ASCIINewline_cell_0_1_1 LFNewline_cell_0_1_2 0ANewline_cell_0_1_3 10Newline_cell_0_1_4 \nNewline_cell_0_1_5
Microsoft Windows, DOS (MS-DOS, PC DOS, etc.), Atari TOS, DEC TOPS-10, RT-11, CP/M, MP/M, OS/2, Symbian OS, Palm OS, Amstrad CPC, and most other early non-Unix and non-IBM operating systemsNewline_header_cell_0_2_0 CR LFNewline_cell_0_2_1 0D 0ANewline_cell_0_2_2 13 10Newline_cell_0_2_3 \r\nNewline_cell_0_2_4
Commodore 8-bit machines (C64, C128), Acorn BBC, ZX Spectrum, TRS-80, Apple II series, Oberon, the classic Mac OS, MIT Lisp Machine and OS-9Newline_header_cell_0_3_0 CRNewline_cell_0_3_1 0DNewline_cell_0_3_2 13Newline_cell_0_3_3 \rNewline_cell_0_3_4
QNX pre-POSIX implementation (version < 4)Newline_header_cell_0_4_0 RSNewline_cell_0_4_1 1ENewline_cell_0_4_2 30Newline_cell_0_4_3 \036Newline_cell_0_4_4
Acorn BBC and RISC OS spooled text output.Newline_header_cell_0_5_0 LF CRNewline_cell_0_5_1 0A 0DNewline_cell_0_5_2 10 13Newline_cell_0_5_3 \n\rNewline_cell_0_5_4
Atari 8-bit machinesNewline_header_cell_0_6_0 ATASCIINewline_cell_0_6_1 Newline_cell_0_6_2 9BNewline_cell_0_6_3 155Newline_cell_0_6_4 Newline_cell_0_6_5
IBM mainframe systems, including z/OS (OS/390) and i5/OS (OS/400)Newline_header_cell_0_7_0 EBCDICNewline_cell_0_7_1 NLNewline_cell_0_7_2 15Newline_cell_0_7_3 21Newline_cell_0_7_4 \025Newline_cell_0_7_5
ZX80 and ZX81 (Home computers from Sinclair Research Ltd)Newline_header_cell_0_8_0 used a specific non-ASCII character setNewline_cell_0_8_1 NEWLINENewline_cell_0_8_2 76Newline_cell_0_8_3 118Newline_cell_0_8_4 Newline_cell_0_8_5

Newline_unordered_list_0

  • EBCDIC systems—mainly IBM mainframe systems, including z/OS (OS/390) and i5/OS (OS/400)—use NL (New Line, 0x15) as the character combining the functions of line feed and carriage return. The equivalent Unicode character (0x85) is called NEL (Next Line). EBCDIC also has control characters called CR and LF, but the numerical value of LF (0x25) differs from the one used by ASCII (0x0A). Additionally, some EBCDIC variants also use NL but assign a different numeric code to the character. However, those operating systems use a , which stores text files as one record per line. In most file formats, no line terminators are actually stored.Newline_item_0_0
  • Operating systems for the CDC 6000 series defined a newline as two or more zero-valued six-bit characters at the end of a 60-bit word. Some configurations also defined a zero-valued character as a colon character, with the result that multiple colons could be interpreted as a newline depending on position.Newline_item_0_1
  • RSX-11 and OpenVMS also use a record-based file system, which stores text files as one record per line. In most file formats, no line terminators are actually stored, but the Record Management Services facility can transparently add a terminator to each line when it is retrieved by an application. The records themselves could contain the same line terminator characters, which could either be considered a feature or a nuisance depending on the application. RMS not only stored records, but also stored metadata about the record separators in different bits for the file to complicate matters even more (since files could have fixed length records, records that were prefixed by a count or records that were terminated by a specific character). The bits weren't generic, so while they could specify that CRLF or LF or even CR was the line terminator, it couldn't substitute some other code.Newline_item_0_2
  • Fixed line length was used by some early mainframe operating systems. In such a system, an implicit end-of-line was assumed every 72 or 80 characters, for example. No newline character was stored. If a file was imported from the outside world, lines shorter than the line length had to be padded with spaces, while lines longer than the line length had to be truncated. This mimicked the use of punched cards, on which each line was stored on a separate card, usually with 80 columns on each card, often with sequence numbers in columns 73–80. Many of these systems added a carriage control character to the start of the next record; this could indicate whether the next record was a continuation of the line started by the previous record, or a new line, or should overprint the previous line (similar to a CR). Often this was a normal printing character such as # that thus could not be used as the first character in a line. Some early line printers interpreted these characters directly in the records sent to them.Newline_item_0_3

Unicode Newline_section_2

"Paragraph separator" redirects here. Newline_sentence_33

For the symbol also known as a "paragraph sign", see Pilcrow. Newline_sentence_34

The Unicode standard defines a number of characters that conforming applications should recognize as line terminators: Newline_sentence_35

Newline_description_list_1

  • LF:    Line Feed, U+000ANewline_item_1_4
  • VT:    Vertical Tab, U+000BNewline_item_1_5
  • FF:    Form Feed, U+000CNewline_item_1_6
  • CR:    Carriage Return, U+000DNewline_item_1_7
  • CR+LF: CR (U+000D) followed by LF (U+000A)Newline_item_1_8
  • NEL:   Next Line, U+0085Newline_item_1_9
  • LS:    Line Separator, U+2028Newline_item_1_10
  • PS:    Paragraph Separator, U+2029Newline_item_1_11

This may seem overly complicated compared to an approach such as converting all line terminators to a single character, for example LF. Newline_sentence_36

However, Unicode was designed to preserve all information when converting a text file from any existing encoding to Unicode and back. Newline_sentence_37

Therefore, Unicode should contain characters included in existing encodings. Newline_sentence_38

NL is included in EBCDIC with code 0x15, and often mapped to NEL, which is a control character in the C1 control set. Newline_sentence_39

As such, it is defined by ECMA 48, and recognized by encodings compliant with ISO/IEC 2022 (which is equivalent to ECMA 35). Newline_sentence_40

C1 control set is also compatible with ISO-8859-1. Newline_sentence_41

The approach taken in the Unicode standard allows round-trip transformation to be information-preserving while still enabling applications to recognize all possible types of line terminators. Newline_sentence_42

Recognizing and using the newline codes greater than 0x7F (NEL, LS and PS) is not often done. Newline_sentence_43

They are multiple bytes in UTF-8, and the code for NEL has been used as the ellipsis (…) character in Windows-1252. Newline_sentence_44

For instance: Newline_sentence_45

Newline_unordered_list_2

  • ECMAScript accepts LS and PS as line-breaks, but considers U+0085 (NEL) whitespace instead of a line-break.Newline_item_2_12
  • Windows 10 does not treat any of NEL, LS, or PS as line-breaks in its default text editor, Notepad.Newline_item_2_13
  • gedit, the default text editor of the GNOME desktop environment, treats LS and PS as newlines but does not for NEL.Newline_item_2_14
  • JSON allows LS and PS characters within strings, while ECMAScript prior to ES2019 treated them as newlines, and therefore illegal syntax.Newline_item_2_15
  • YAML no longer recognizes them as special as of version 1.2, in order to be compatible with JSON.Newline_item_2_16

The Unicode characters U+2424 (SYMBOL FOR NEWLINE, ␤), U+23CE (RETURN SYMBOL, ⏎), U+240D (SYMBOL FOR CARRIAGE RETURN, ␍) and U+240A (SYMBOL FOR LINE FEED, ␊) are intended for presenting a user-visible character to the reader of the document, and are thus not recognized themselves as a newline. Newline_sentence_46

Escape sequences Newline_section_3

An escape sequence is a combination of characters which represents no text; instead of being displayed (as text) it is supposed to be intercepted by the program and a special function is supposed to be performed. Newline_sentence_47

Escape sequences are also used to handle (set, search, replace, etc.) special characters. Newline_sentence_48

In programming languages Newline_section_4

"\n" redirects here. Newline_sentence_49

For the similar number notation, see \nnn (disambiguation). Newline_sentence_50

To facilitate the creation of portable programs, programming languages provide some abstractions to deal with the different types of newline sequences used in different environments. Newline_sentence_51

The C programming language provides the escape sequences '\n' (newline) and '\r' (carriage return). Newline_sentence_52

However, these are not required to be equivalent to the ASCII LF and CR control characters. Newline_sentence_53

The C standard only guarantees two things: Newline_sentence_54

Newline_ordered_list_3

  1. Each of these escape sequences maps to a unique implementation-defined number that can be stored in a single char value.Newline_item_3_17
  2. When writing to a file, device node, or socket/fifo in text mode, '\n' is transparently translated to the native newline sequence used by the system, which may be longer than one character. When reading in text mode, the native newline sequence is translated back to '\n'. In binary mode, no translation is performed, and the internal representation produced by '\n' is output directly.Newline_item_3_18

On Unix platforms, where C originated, the native newline sequence is ASCII LF (0x0A), so '\n' was simply defined to be that value. Newline_sentence_55

With the internal and external representation being identical, the translation performed in text mode is a no-op, and Unix has no notion of text mode or binary mode. Newline_sentence_56

This has caused many programmers who developed their software on Unix systems simply to ignore the distinction completely, resulting in code that is not portable to different platforms. Newline_sentence_57

The C library function fgets() is best avoided in binary mode because any file not written with the Unix newline convention will be misread. Newline_sentence_58

Also, in text mode, any file not written with the system's native newline sequence (such as a file created on a Unix system, then copied to a Windows system) will be misread as well. Newline_sentence_59

Another common problem is the use of '\n' when communicating using an Internet protocol that mandates the use of ASCII CR+LF for ending lines. Newline_sentence_60

Writing '\n' to a text mode stream works correctly on Windows systems, but produces only LF on Unix, and something completely different on more exotic systems. Newline_sentence_61

Using "\r\n" in binary mode is slightly better. Newline_sentence_62

Many languages, such as C++, Perl, and Haskell provide the same interpretation of '\n' as C. C++ has an alternative I/O model where the manipulator std::endl can be used to output a newline (and flushes the stream buffer). Newline_sentence_63

Java, PHP, and Python provide the '\r\n' sequence (for ASCII CR+LF). Newline_sentence_64

In contrast to C, these are guaranteed to represent the values U+000D and U+000A, respectively. Newline_sentence_65

The Java I/O libraries do not transparently translate these into platform-dependent newline sequences on input or output. Newline_sentence_66

Instead, they provide functions for writing a full line that automatically add the native newline sequence, and functions for reading lines that accept any of CR, LF, or CR+LF as a line terminator (see ). Newline_sentence_67

The method can be used to retrieve the underlying line separator. Newline_sentence_68

Example: Newline_sentence_69

Python permits "Universal Newline Support" when opening a file for reading, when importing modules, and when executing a file. Newline_sentence_70

Some languages have created special variables, constants, and subroutines to facilitate newlines during program execution. Newline_sentence_71

In some languages such as PHP and Perl, double quotes are required to perform escape substitution for all escape sequences, including '\n' and '\r'. Newline_sentence_72

In PHP, to avoid portability problems, newline sequences should be issued using the PHP_EOL constant. Newline_sentence_73

Example in C#: Newline_sentence_74

Issues with different newline formats Newline_section_5

Even though the control characters are unambiguously defined in the corresponding character encoding table used by a text file, there still is an issue: there are different conventions to set and display a line break. Newline_sentence_75

To denote a single line break, Unix programs use line feed, whose hexadecimal value in ASCII is 0a, while most programs common to MS-DOS and Microsoft Windows use carriage return+line feed, whose hexadecimal value in ASCII is 0d 0a. Newline_sentence_76

In ASCII, carriage return is a distinct control character. Newline_sentence_77

The different newline conventions cause text files that have been transferred between systems of different types to be displayed incorrectly. Newline_sentence_78

Text in files created with programs which are common on Unix-like or classic Mac OS, appear as a single long line on most programs common to MS-DOS and Microsoft Windows because these do not display a single line feed or a single carriage return as a line break. Newline_sentence_79

Conversely, when viewing a file originating from a Windows computer on a Unix-like system, the extra CR may be displayed as a second line break, as ^M, or as <cr> at the end of each line. Newline_sentence_80

Furthermore, programs other than text editors may not accept a file, e.g. some configuration file, encoded using the foreign newline convention, as a valid file. Newline_sentence_81

The problem can be hard to spot because some programs handle the foreign newlines properly while others do not. Newline_sentence_82

For example, a compiler may fail with obscure syntax errors even though the source file looks correct when displayed on the console or in an editor. Newline_sentence_83

On a Unix-like system, the command cat -v myfile.txt will send the file to stdout (normally the terminal) and make the ^M visible, which can be useful for debugging. Newline_sentence_84

Modern text editors generally recognize all flavours of CR+LF newlines and allow users to convert between the different standards. Newline_sentence_85

Web browsers are usually also capable of displaying text files and websites which use different types of newlines. Newline_sentence_86

Even if a program supports different newline conventions, these features are often not sufficiently labeled, described, or documented. Newline_sentence_87

Typically a menu or combo-box enumerating different newline conventions will be displayed to users without an indication if the selection will re-interpret, temporarily convert, or permanently convert the newlines. Newline_sentence_88

Some programs will implicitly convert on open, copy, paste, or save—often inconsistently. Newline_sentence_89

Most textual Internet protocols (including HTTP, SMTP, , IRC, and many others) mandate the use of ASCII CR+LF ('\r\n', 0x0D 0x0A) on the protocol level, but recommend that tolerant applications recognize lone LF ('\n', 0x0A) as well. Newline_sentence_90

Despite the dictated standard, many applications erroneously use the C newline escape sequence '\n' (LF) instead of the correct combination of carriage return escape and newline escape sequences '\r\n' (CR+LF) (see section Newline in programming languages above). Newline_sentence_91

This accidental use of the wrong escape sequences leads to problems when trying to communicate with systems adhering to the stricter interpretation of the standards instead of the suggested tolerant interpretation. Newline_sentence_92

One such intolerant system is the qmail mail transfer agent that actively refuses to accept messages from systems that send bare LF instead of the required CR+LF. Newline_sentence_93

The standard Internet Message Format for eMail states: CR and LF MUST only occur together as CRLF; they MUST NOT appear independently in the body. Newline_sentence_94

The can automatically convert newlines in files being transferred between systems with different newline representations when the transfer is done in "ASCII mode". Newline_sentence_95

However, transferring binary files in this mode usually has disastrous results: any occurrence of the newline byte sequence—which does not have line terminator semantics in this context, but is just part of a normal sequence of bytes—will be translated to whatever newline representation the other system uses, effectively corrupting the file. Newline_sentence_96

FTP clients often employ some heuristics (for example, inspection of ) to automatically select either binary or ASCII mode, but in the end it is up to users to make sure their files are transferred in the correct mode. Newline_sentence_97

If there is any doubt as to the correct mode, binary mode should be used, as then no files will be altered by FTP, though they may display incorrectly. Newline_sentence_98

Conversion between newline formats Newline_section_6

Text editors are often used for converting a text file between different newline formats; most modern editors can read and write files using at least the different ASCII CR/LF conventions. Newline_sentence_99

For example, the editor Vim can make a file compatible with the Windows Notepad text editor. Newline_sentence_100

Within vim Newline_sentence_101

Editors can be unsuitable for converting larger files or bulk conversion of many files. Newline_sentence_102

For larger files (on Windows NT/2000/XP) the following command is often used: Newline_sentence_103

Special purpose programs to convert files between different newline conventions include unix2dos and dos2unix, mac2unix and unix2mac, mac2dos and dos2mac, and flip. Newline_sentence_104

The tr command is available on virtually every Unix-like system and can be used to perform arbitrary replacement operations on single characters. Newline_sentence_105

A DOS/Windows text file can be converted to Unix format by simply removing all ASCII CR characters with Newline_sentence_106

or, if the text has only CR newlines, by converting all CR newlines to LF with Newline_sentence_107

The same tasks are sometimes performed with awk, sed, or in Perl if the platform has a Perl interpreter: Newline_sentence_108

The command can identify the type of line endings: Newline_sentence_109

The Unix egrep (extended grep) command can be used to print filenames of Unix or DOS files (assuming Unix and DOS-style files only, no Mac OS): Newline_sentence_110

Other tools permit the user to visualise the EOL characters: Newline_sentence_111

Interpretation Newline_section_7

Two ways to view newlines, both of which are self-consistent, are that newlines either separate lines or that they terminate lines. Newline_sentence_112

If a newline is considered a separator, there will be no newline after the last line of a file. Newline_sentence_113

Some programs have problems processing the last line of a file if it is not terminated by a newline. Newline_sentence_114

On the other hand, programs that expect newline to be used as a separator will interpret a final newline as starting a new (empty) line. Newline_sentence_115

Conversely, if a newline is considered a terminator, all text lines including the last are expected to be terminated by a newline. Newline_sentence_116

If the final character sequence in a text file is not a newline, the final line of the file may be considered to be an improper or incomplete text line, or the file may be considered to be improperly truncated. Newline_sentence_117

In text intended primarily to be read by humans using software which implements the word wrap feature, a newline character typically only needs to be stored if a line break is required independent of whether the next word would fit on the same line, such as between paragraphs and in vertical lists. Newline_sentence_118

Therefore, in the logic of word processing and most text editors, newline is used as a paragraph break and is known as a "hard return", in contrast to "soft returns" which are dynamically created to implement word wrapping and are changeable with each display instance. Newline_sentence_119

In many applications a separate control character called "manual line break" exists for forcing line breaks inside a single paragraph. Newline_sentence_120

The glyph for the control character for a hard return is usually a pilcrow (¶), and for the manual line break is usually a carriage return arrow (↵). Newline_sentence_121

Reverse and partial line feeds Newline_section_8

See also Newline_section_9

Newline_unordered_list_4


Credits to the contents of this page go to the authors of the corresponding Wikipedia page: en.wikipedia.org/wiki/Newline.