file-based integration

Core concept

File-based integration is a method of exchanging data between systems by creating, transferring, and consuming files rather than calling APIs or sharing databases directly. One system writes data to a file in an agreed format, that file is moved or made accessible to another system, and the receiving system reads and processes the contents.

File-based integration commonly uses formats such as CSV, XML, JSON, or fixed-width text, and transfer mechanisms such as shared network folders, SFTP, or managed file transfer tools.

How it works in industrial and regulated environments

In manufacturing and other industrial operations, file-based integration is frequently used to connect:

– MES and ERP systems (e.g., production orders, material consumption, inventory, batch records)
– LIMS, quality systems, and data historians to higher-level business systems
– Legacy equipment or OT applications that can only export or import data via files

Typical patterns include:

– **Outbound exports**: A source system periodically generates a file (for example, all production confirmations since the last run) and places it in a predefined location for the target system to pick up.
– **Inbound imports**: A system monitors a directory or endpoint for new files, then validates, parses, and loads the data into internal tables or queues.
– **Staging and archiving**: Files are often staged in intermediate folders (e.g., `inbound`, `processing`, `error`, `archive`) to support traceability, troubleshooting, and reprocessing.

In regulated environments, file-based integration is sometimes favored for specific interfaces because it is:

– Easier to trace and archive (files provide a clear, inspectable record of what was exchanged)
– Compatible with legacy systems that do not support APIs or messaging
– Operable in environments with strict network segmentation, where direct service calls are limited

Boundaries and exclusions

File-based integration:

– **Includes** any integration pattern where the primary contract between systems is a file (format + structure + location + timing), regardless of transport protocol.
– **Does not require** a particular technology stack; it can be implemented with simple scripts, ETL tools, ESB/iPaaS platforms, or custom applications.
– **Does not mean** that data is unstructured; file contents are often highly structured and governed by strict schemas.

It is distinct from:

– **API/web service integration**, where systems communicate via synchronous or asynchronous service calls rather than exchanging files.
– **Direct database integration**, where one system reads from or writes to another system’s database or shared views.
– **Message-queue integration**, where messages are pushed to and consumed from a broker without relying on file artifacts as the primary contract.

Common usage and considerations

In practice, file-based integrations are typically:

– **Batch-oriented**: Data is exchanged at defined intervals (e.g., every 5 minutes, hourly, end-of-shift) rather than continuously.
– **Interface-specific**: Each file type represents a specific business object or transaction set (e.g., `PRODUCTION_ORDER`, `GOODS_MOVEMENT`, `QC_RESULT`).
– **Schema-controlled**: Changes to file structure (fields, delimiters, encoding) usually require coordinated updates and testing in all participating systems.

File-based integrations are often combined with other patterns in hybrid architectures. For example, an MES may exchange master data with ERP via files while using web services for near-real-time order confirmations.

Relation to MES–ERP integration (site context)

Within MES–ERP integration, file-based integration commonly refers to exports and imports of transactional and master data files between the two systems. Examples include:

– ERP generating planned orders as files that the MES imports and converts into executable work orders
– MES sending production confirmations, material consumption, and quality result files back to ERP on a schedule
– Use of controlled share folders or SFTP endpoints managed by IT or an ESB/iPaaS as the transfer channel

In this context, file-based integration is one of several standard patterns, alongside database views, APIs/web services, and message queues, chosen based on system capabilities, validation impact, and operational constraints.