Insights
Check the file before the load: pre-ingestion rules for delimited data.
A surprising share of failed data loads come from files with the wrong header, a shifted column, or an uneven row, discovered only after an ingestion job fails at two in the morning. The checks below take minutes and move the problem back to the sender, where it is cheap to fix. FileSpecGuard, a free browser-local tool from Ascendant Systems, runs them without the file leaving the machine.
the checks
The checks that catch most failures
- Header presence and exact names: every expected column exists and nothing unexpected has appeared.
- Column order: the order matches the specification when the loader is positional.
- Row width: every row has the same number of fields as the header, which catches unquoted delimiters and broken line endings.
- Delimiter and quoting: the declared delimiter is the one used, and quoted fields close.
- Encoding and line endings: UTF-8 without a stray byte-order mark, and consistent line endings.
- Required fields: mandatory columns are populated on every row.
why with the sender
Why the check belongs with the sender
When the receiving system rejects a file, the fix requires a ticket, a conversation, and a resend, often a day later. When the sender runs the same rules before sending, the fix takes a minute and no one else is involved. Publishing the rules as a checklist the sender can run is the highest-leverage data-quality control most teams have not implemented.
keep data local
Keep the data where it is
Files exchanged between organizations frequently contain personal, financial, or health information. A structure check does not need the values, only the shape. FileSpecGuard runs entirely in the browser, keeps the file on the device, and produces a report of findings that contains no cell values, so it can be shared with the receiving team without a data-sharing review.
rules as contract
Treat the rules as the interface contract
Write the header list, order, required fields, and delimiter down once, version it, and give both sides the same copy. When the specification changes, change the document first and the pipeline second. Most file-exchange incidents trace to a specification that lived only in the loader's code.
putting it in place
Putting the check in place with a partner
Rolling this out takes one short conversation per sender. Share the written specification and the tool link, ask the sender to run the check before each delivery, and add the step to the partner onboarding checklist so new senders inherit it. Most senders accept immediately, because a rejected file costs them a resend and a delayed payment or report.
- Send the specification and the check together, never the specification alone.
- Ask for the findings report, not the file, when a delivery fails; it contains no cell values and shows exactly which rule failed.
- Log every rejection with the rule that caught it, so recurring sender problems become visible and fixable at the source.
- Review the rejection log quarterly with the sender and retire rules that no longer catch anything.
beyond structure
What structure checks do not cover
A well-formed file can still carry wrong values: dates in the wrong format, amounts in the wrong unit, codes outside the allowed set. Those belong to the next layer, validation inside the pipeline with quarantine and reason codes. Ascendant Systems builds both layers as part of data engineering engagements, with the structure rules published to senders and the value rules enforced at load.
Next step
Bring the problem behind the article
If one of these situations looks like yours, describe the outcome you need and what is getting in the way. The first exchange decides whether a responsible scope can be defined.