Skip to main content

Read files from an SFTP or FTP folder

A folder source signs in to a server the sending system writes files to, reads new files every few minutes, and moves each file out of the way once read. Laboratories often send results this way.

Before you begin​

Have the following ready:

  1. The server's host name and port (SFTP usually 22, FTP 21).
  2. A user name and either a password or, for SFTP, a private key. Read access to the folder, and permission to move files within it, is enough.
  3. The folder the files arrive in and what the file names look like, for example *.hl7.
  4. What the files contain:
    • HL7 v2 (one or many messages per file), including HL7 batch files (FHS/BHS … BTS/FTS).
    • CSV (one message per row).
    • Pipe-delimited (one message per row).
  5. The admin role in the workspace.

1. Add the folder​

  1. Open Sources and select New source.
  2. Select SFTP folder or FTP folder.
  3. Enter a Name, for example Pathology results folder, and choose the feed under Send messages into, for example HL7 ORU -> FHIR Bundle (Scenario 2) for results.
  4. Enter the Host, Port and User name.
  5. Sign in:
    • SFTP: under Sign in with choose Password or Private key, then enter it. A key can have a passphrase.
    • FTP: enter the Password, and tick Use TLS (FTPS) if the server supports it.
  6. Enter the Folder to read, the File names to pick up, and What the files contain.
  7. Optionally change where files go once read (Move read files to, default processed) or when they cannot be read (Move unreadable files to, default failed), how often to poll, and how many files to read per poll.
  8. Select Add source.

New SFTP folder source

Use TLS for FTP

Without TLS, the FTP password and the messages cross the network in clear text. Prefer SFTP, or FTP with Use TLS (FTPS) ticked.

2. Test the connection​

Select Test connection. NEEV signs in, lists the folder and says how many files are waiting, for example "Connected to sftp.pathology-lab.example as neev-reader. No files waiting in /outbound/results."

For SFTP, the first connection also records the server's key fingerprint. It is shown under Server key. If a different server ever answers at that address, polling stops and says so, instead of handing patient files to it: "The server's identity has changed since it was first trusted…", and the source shows Could not read.

If the change is expected, for example the server was rebuilt, confirm the new fingerprint with whoever runs the server. Then an admin selects Forget under Server key and Test connection, which trusts the key the server shows now. Polling picks up where it stopped; no file is skipped.

3. Switch it on and read files​

  1. Switch the source on. It polls every 5 minutes unless you changed it.
  2. To read straight away, select Poll now.

One file read: three messages, all accepted

The panel shows the result of the last poll, for example "Read 1 file: 3 messages, 3 accepted.", and Recent polls lists each file with how many messages it held and where it was moved.

How files are read​

  • Each message goes through on its own. A batch file's headers and trailers are dropped and each message in it is sent to the feed separately. A CSV file sends one message per row.
  • Files still being written are skipped. Names ending .tmp or .part are left alone until the sender renames them.
  • Nothing is read twice. A file that was read is moved to the processed folder. If the same file appears again, it is moved out without being sent again ("Read before and not sent again").
  • A file is read whole or not at all. A file that cannot be split cleanly, such as one with no MSH segment, is moved to the failed folder with the reason, and the source counts under Need a look. Nothing in it is sent, so no message is half-delivered.