Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[csv-loader] Support for custom dynamicTyping option #50

Open
marceloverdijk opened this issue Oct 25, 2024 · 2 comments
Open

[csv-loader] Support for custom dynamicTyping option #50

marceloverdijk opened this issue Oct 25, 2024 · 2 comments

Comments

@marceloverdijk
Copy link

marceloverdijk commented Oct 25, 2024

When loading the CSV the papaparse dynamicTyping seems always be set to true, and cannot be overridden:

https://github.com/ascorbic/astro-loaders/blob/main/packages/csv/src/csv-loader.ts#L55C1-L60C8

    const csvStream = Papa.parse(Papa.NODE_STREAM_INPUT, {
      dynamicTyping: true,
      ...parserOptions,
      header: true,
      transformHeader: transformHeader === false ? undefined : transformHeader,
    });

parserOptions?: Omit<
    Papa.ParseConfig,
    "header" | "dynamicTyping" | "transformHeader" | "step" | "complete"
  >;

(same for header and transformHeader, step and complete).

The issue is I have a CSV like:

"name","category","priority"
"foo","A1",1
"bar","A2",2 
"xyz","1",1 

Where category is a string (all values are quoted as well),
but when parsing "1" it still converts it to a number (I believe because of dynamicTyping = true) which result in a terminal error:

AstroError [InvalidContentEntryDataError]: **persons → xyz** data does not match collection schema.
**category**: Expected type `"string"`, received "number"

I think in this case I should configure the csv loader like:

const persons = defineCollection({
  loader: csvLoader({
    fileName: 'data/persons.csv',
    transformHeader: false,
    idField: 'name',
    parserOptions: {
      dynamicTyping: (field) => field !== 'category',
    },
  }),
  schema: z.object({
    name: z.string(),
    category: z.string(),
    priority: z.number().int(),
  }),
});
@ascorbic
Copy link
Owner

Yes, I think something analogous to the glob loader's generateId function would make sense here.

@marceloverdijk
Copy link
Author

@ascorbic I had my data in both json and csv format and I think using the out-of-the-box file loader to load the JSON might be a better solution than using csv.

However the file loader also the limitation that it cannot generate ids.

After you mentioned the glob's generateId option I changed the file loader my self to be able to generate ids, and that works perfectly.

I've created a feature request and a PR here

withastro/roadmap#1045

withastro/astro#12308

I'm looking forward to your feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants