[Go] array.RecordFromJSON() method does not return for a significant amount of time for large JSON file

[AlwaysZhi]

Describe the usage question you have. Please include as many useful details as possible.

When I attempt to use the array.RecordFromJSON() method to read a relatively large JSON file (approximately 200MB in size), Ithe method does not return for a significant amount of time. It seems to be either blocked indefinitely or possibly stuck in an ongoing data processing loop. I am currently unable to determine whether this is a bug or how to successfully utilize the method to achieve my objective.
Here is my code:

func readJsonFile() {
schema := arrow.NewSchema([]arrow.Field{
{Name: "instruction", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "input", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "output", Type: arrow.BinaryTypes.String, Nullable: true},
}, nil)
jsonFile, err := os.Open("/home/data/mydata.json")
if err != nil {
fmt.Println("open json file:", err)
return
}
defer jsonFile.Close()
mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
rec, _, err := array.RecordFromJSON(mem, schema, jsonFile)
fmt.Printf("record number is： %d", rec.NumRows())
fmt.Println("field name 0 -", rec.Schema().Field(0).Name)
fmt.Println("field name 1 -", rec.Schema().Field(1).Name)
fmt.Println("field name 2 -", rec.Schema().Field(2).Name)
}

My json file format looks like this:
[{"instruction":"a", "input":"b", "output","o1"}, {"instruction":"c", "input":"d", "output","o2"}, ...]

Component(s)

Go

[zeroshade]

I'll take a look at this sometime this week.

[zeroshade]

Would you be able to provide the actual data file that causes the problem? I created a 200MB JSON file with random values using the format schema you provided and then ran the code you provided to read it. It took just around 7 seconds or so to run and I wasn't able to reproduce indefinite blocking that you reported.

[AlwaysZhi]

ok， here is my data file
data.zip

[AlwaysZhi]

Now i convert the json format and use WithMultipleDocs option, it works well.

func TestReadJson(t *testing.T) {
	file, err := os.Open("/home/data/180000.json")
	if err != nil {
		fmt.Println("open json:", err)
	}
	defer file.Close()
	content, err := io.ReadAll(file)
	if err != nil {
		fmt.Println("read file:", err)
	}

	var jsonData []map[string]interface{}
	err = json.Unmarshal(content, &jsonData)
	if err != nil {
		fmt.Println("Unmarshal failed:", err)
	}
	schema := arrow.NewSchema([]arrow.Field{
		{Name: "instruction", Type: arrow.BinaryTypes.String, Nullable: true},
		{Name: "input", Type: arrow.BinaryTypes.String, Nullable: true},
		{Name: "output", Type: arrow.BinaryTypes.String, Nullable: true},
	}, nil)
	mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
	var recs []arrow.Record
	for _, obj := range jsonData {
		bytes, err := json.Marshal(obj)
		if err != nil {
			fmt.Println("Marshal failed:", err)
		}
		rec, _, err := array.RecordFromJSON(mem, schema, strings.NewReader(string(bytes)), array.WithMultipleDocs())
		recs = append(recs, rec)
	}
	fmt.Println("value -", recs[0].Column(0))
	fmt.Println("value -", recs[0].Column(1))
	fmt.Println("value -", recs[0].Column(2))
}

[zeroshade]

@AlwaysZhi was using the WithMultipleDocs option enough to solve the issue? or is there more of a question here?

Typically you need the WithMultipleDocs option if the format of the json file is:

{"a": 1, "b": 2......}
{"a": 2, "b": 3......}
{.......}

Rather than having them all wrapped inside of [ ] as a single JSON document