💡PapaParse 6.0

[dboskovic]

Would love to get your feedback on my wishlist for Papaparse. We're going to dedicate some time for the team @ Flatfile before the end of year to move anything on this list we're agreed on to a next release.

1. Migrate to ES7 + Typescript or Flow

This is something that, after a deep audit of the source code we feel is necessary to ensure long term reliability. In a data oriented library like this, type strength will allow for easier reasoning about the code, add stability and prevent unseen bugs. ES7 will allow for more readable code

Since not everybody understands TS or Flow deeply enough to have confidence to contribute. My recommendation is to keep the config permissive to allow for vanillaJS contributions and have core contributors add stronger typing before merging to master.

2. Separate NodeJS build from Browser build

This will allow for a lighter package when using in the browser (vast majority based on cursory analysis), as well as open more freedom to invest in optimizations for each stack independently. These could either be distributed as different packages (eg. @papaparse/core and @papaparse/node) or a second build in the same package. (eg. (import PapaParse from 'papaparse/node')

3. Reduce core sugar and add plugin framework

We've noticed a lot of the open issues relate to desired support of edge cases or unique data scenarios that shouldn't be treated as part of the core "csv parser" but are entirely legitimate use cases. The goal here is to distribute a core package with common functionality and allow users to choose additional use cases as needed.

Candidates:

• @papaparse/http - adapter for downloading or streaming data from web - can be optimized separately for nodejs and browser as well as opens up for other adapters for things like S3 with plenty of optimizations.
• @papaparse/types - split out the typecasting logic, there's a lot of room for improvement here w/better understanding of boolean types, dates, etc. But it doesn't make sense to invest that into a core csv parser library.
• @papaparse/unparse - there's been a decent amount of confusion with users about how different configurations relate to parse vs. unparse. These are also distinctly different problems to solve for.

Future Candidates:
Things like detect-encoding to auto detect file encoding, generous-escaping (for the common unescaped quotes situation), and many other user requests. Additionally framework specific components like an HoC for React could be awesome.

4. Improved docs

Would love to see updated searchable docs with both auto generated API references as well as guides, fiddles, and improved demo. I'm a fan of docusaurus for this. We'd be happy to contribute content & design here.

5. Reorganized source code

With almost 2000 lines in papaparse.js it's time to tackle deconstructing that a bit into components that are easier to reason about. Since 4b16215 6 years ago (335 lines) when most of the tools we have at our disposal today weren't available, we haven't changed much. Time for a src folder! Let's follow https://sourcemaking.com/refactoring as a guide

6. Tests, coverage, cross-browser testing & CI based distribution

We should take advantage of setting up the Sauce testing matrix so we don't break things in old browsers as we go. Also, it'd be great if we could use Github Actions to auto deploy master and release candidates to npm / bower / etc. In addition we should improve unit test coverage in addition to the mainly acceptance testing we have now.

7. Other: Pipes, Promises, etc.

• Piping for easier composition of logic, also relatively required for plugin framework.
• Promises because it's 2019 (shim for legacy browsers)
• Allow for some dependencies and ensure fossa.com scans are run on them - let's not re-invent all the wheels.
• Functional first - no classes unless necessary
• Package decomposition - As seen in the above example, switching to an npm org w/multiple packages and likely using something like lerna to manage the packages.

8. Backwards compatibility adapter

Because this would be a pretty robust overhaul, we should publish an adapter that's fully backwards compatible with re-composed elements. Possibly @papaparse/legacy - allowing people to move forwards without a complete overhaul. It could also identify the things they aren't using and give them a custom migration checklist.

We're happy to take on the work of this overhaul here at Flatfile - so keep in mind we're not asking for a lot of work from the community. But do please provide feedback on all of this, we want to chart a path forwards that makes sense to everybody.

Also, what do you want to see? Comment with new ideas or criticisms / approval of the above.

[caverna]

👍 for Typescript! (or at least an official .d.ts file!)

[pokoli]

Hi @dboskovic,

First of all thank you so much for all the effort put on your issue. I think they are very nice proposals and we should go forward and implement most of them. I have some concers I will like to discuss without going forward:

• About TypeScript I seems good to me as far as we also allow to use PapaParse with vanilla javascript.
• About the plugin I'm not sure if this will be to much enginering for a CSV Parsing library. Maybe we should implement first one of the proposed plugins to have a good structure and the implement the others.
• Another doubt about the plugin system is that I'm not sure if they will be distrubuted on the same package or included on separate packages. In case the proposal is to create separate packages we need to define where this code wil be managed (on the same repository, on a separate repository) and how the integration should work.

I do not much the idea of a backwards compatible layer. I prefer to have a timeline to support the 5.0x series, so we fix issues on this branch while allowing users to test the new version.

For the documentation we only need to generate some static files in html format than can be latter uploaded to our website, so any dynamism here makes sense. I think that we can change the documentation format without requiring a new version to be released.

For all the other points, please go forward on them and strat creating PR for them!
My idea is to release 5.2.0 before merging breaking changes to master branch.

[DRocksCoding]

Thanks, I'm new to javascript and I was able to read a csv with your website example and your documentation it's good.

I also like the idea of getting core parts available on their own for less complex situations.
Thank you,
Cheers

Alex

[jimmywarting]

es7 + jsdoc and/or d.ts

wish to be able to use native import in browser without having to bundle or transpile the scripts. typescript is not javascript and dose not work in browser.
(and for the record use full path with extension so browser can properly require related files)

[jimmywarting]

if you would like to write cross node/browser maybe you should consider abending streams and use async iterator instead

[maxwell8888]

It would be great to provide an ES module build. Most javascript packages provide both a node.js build, and an ES module build. If you use tools like Webpack or Rollup (recommended) it is simply a case of specifying in the config you would like to additionally output a module build, which is then referenced for example module: es_module/index.jsinpackage.json`. At the moment, I am not able to bundle PapaParse with Rollup. I assume that other users are either not bundling and simply including the browser script, or they are using Webpack, which is the most popular tool, however rollup is the second most popular and very heavily used.

[gunn]

I would like to see support for streaming data from node as it is unparsed as per #568 and #652.

[DaSchTour]

I would like to see support for streaming data from node as it is unparsed as per #568 and #652.

But please implement it in a way that this library still can be used in browser ;)

[MarsianMan]

Promises because it's 2019 (shim for legacy browsers)

Maybe move the milestone to async/await for 2021 😄

[jimmywarting]

I wish you would use a async generator instead of importing node streams
https://github.com/cross-js/cross-js#dont-create-node-or-web-readable-stream-yourself

(or use whatwg stream on nodejs) so it could work grate across Web/Deno/NodeJS