Splitting a Large Text File


Some text files are so large that they become difficult to work with. For example, If you try to open a text file that has more than 1,048,576 rows, Excel will give you a warning using the yellow icon with the exclamation mark. It says “The data set is too large for the Excel grid. If you save this workbook you’ll lose data that wasn’t loaded.” Older versions of Excel have maximum lines/row limits that are much fewer. To open and view a large text file you might try the program Large Text Viewer. It’s not just about viewing files though. Perhaps you need to import a text file into a database program like SQL Server. While SQL Server may be able to handle it, does your computer have enough RAM? If your file is several hundred GB, SQL Server may give you an out-of-memory error and be unable to import the file into a table. The solution is to split the file into several smaller files and import those, then merge them.

If you have installed Git for Windows, you should have Git Bash installed, since that comes with Git. You can use the split command in Git Bash to split a file. In the second example, that’s a dash and the letter l.

  • Split a file into files of size 300MB each: split bigFile.txt -b 300m
  • Split a file into files with 1 million lines each: split bigFile.txt -l 1000000

If you are going import lines of data, into a database for example, the second example where you specify the number of lines will work better.

In Windows, you can use File Explorer to open a Git Bash command prompt at the location of your large file by using the right-click popup menu and choosing Git Bash Here. Now you can just go ahead with your command and Bash will be able to find you file.

Files will be created with the following name patter: xaa, xab, xac and so on. Alternatively you can specify a prefix and numerical suffix. Please the the Stack Overflow post called How to split large text file in windows and read the second part of the first answer.