Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Import grading report #27

Open
lamnguyen8075 opened this issue Aug 5, 2021 · 2 comments
Open

Import grading report #27

lamnguyen8075 opened this issue Aug 5, 2021 · 2 comments
Assignees
@phuocdn
Labels
bug Something isn't working

Comments

@lamnguyen8075
Copy link

lamnguyen8075 commented Aug 5, 2021 •
edited
Loading

Current behavior:

Expected behavior:
@lamnguyen8075 lamnguyen8075 added the bug Something isn't working label Aug 5, 2021
@phuocdn
Copy link

phuocdn commented Sep 29, 2021 •
edited
Loading

Error logs:

2021-09-29 08:29:09,597 INFO 2701 [staff_graded.staff_graded] [user 25994] staff_graded.py:201 - Processing 2276 byte score file aoc.csv for block-v1:STEAMforVietnam+Scratch100_Testing+2020-08+type@staffgradedxblock+block@b9ccdab206f84fbfa43e0aa45e92e62d
2021-09-29 08:29:09,651 ERROR 2701 [lms.djangoapps.courseware.module_render] [user 25994] module_render.py:1221 - error executing xblock handler
Traceback (most recent call last):
  File "/openedx/edx-platform/lms/djangoapps/courseware/module_render.py", line 1196, in _invoke_xblock_handler
    resp = handler_instance.handle(handler, req, suffix)
  File "/openedx/venv/lib/python3.6/site-packages/xblock/mixins.py", line 89, in handle
    return self.runtime.handle(self, handler_name, request, suffix)
  File "/openedx/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1453, in handle
    return super(MetricsMixin, self).handle(block, handler_name, request, suffix=suffix)
  File "/openedx/venv/lib/python3.6/site-packages/xblock/runtime.py", line 1063, in handle
    results = handler(request, suffix)
  File "/openedx/venv/lib/python3.6/site-packages/staff_graded/staff_graded.py", line 208, in csv_import_handler
    processor.process_file(score_file, autocommit=True)
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/csv_processor.py", line 187, in process_file
    self.preprocess_file(reader)
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/mixins.py", line 165, in preprocess_file
    operation = self.save('error')
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/mixins.py", line 123, in save
    user=operating_user or get_current_user(),
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/models.py", line 78, in record_operation
    instance.data.save(uuid.uuid4(), ContentFile(data))
  File "/openedx/venv/lib/python3.6/site-packages/django/db/models/fields/files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/openedx/venv/lib/python3.6/site-packages/django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "/openedx/venv/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/openedx/venv/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/openedx/venv/lib/python3.6/site-packages/boto3/s3/inject.py", line 513, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/openedx/venv/lib/python3.6/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/client.py", line 596, in _make_api_call
    request_signer=self._request_signer, context=request_context)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/hooks.py", line 242, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/handlers.py", line 209, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/handlers.py", line 187, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/handlers.py", line 201, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing
2021-09-29 08:29:09,656 ERROR 2701 [root] [user None] signals.py:23 - Uncaught exception from None
Traceback (most recent call last):
  File "/openedx/venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/openedx/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/openedx/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/django/views/decorators/clickjacking.py", line 50, in wrapped_view
    resp = view_func(*args, **kwargs)
  File "/openedx/edx-platform/lms/djangoapps/courseware/module_render.py", line 1072, in handle_xblock_callback
    return _invoke_xblock_handler(request, course_id, usage_id, handler, suffix, course=course)
  File "/openedx/edx-platform/lms/djangoapps/courseware/module_render.py", line 1196, in _invoke_xblock_handler
    resp = handler_instance.handle(handler, req, suffix)
  File "/openedx/venv/lib/python3.6/site-packages/xblock/mixins.py", line 89, in handle
    return self.runtime.handle(self, handler_name, request, suffix)
  File "/openedx/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1453, in handle
    return super(MetricsMixin, self).handle(block, handler_name, request, suffix=suffix)
  File "/openedx/venv/lib/python3.6/site-packages/xblock/runtime.py", line 1063, in handle
    results = handler(request, suffix)
  File "/openedx/venv/lib/python3.6/site-packages/staff_graded/staff_graded.py", line 208, in csv_import_handler
    processor.process_file(score_file, autocommit=True)
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/csv_processor.py", line 187, in process_file
    self.preprocess_file(reader)
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/mixins.py", line 165, in preprocess_file
    operation = self.save('error')
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/mixins.py", line 123, in save
    user=operating_user or get_current_user(),
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/models.py", line 78, in record_operation
    instance.data.save(uuid.uuid4(), ContentFile(data))
  File "/openedx/venv/lib/python3.6/site-packages/django/db/models/fields/files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/openedx/venv/lib/python3.6/site-packages/django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "/openedx/venv/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/openedx/venv/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/openedx/venv/lib/python3.6/site-packages/boto3/s3/inject.py", line 513, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/openedx/venv/lib/python3.6/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/client.py", line 596, in _make_api_call
    request_signer=self._request_signer, context=request_context)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/hooks.py", line 242, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/handlers.py", line 209, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/handlers.py", line 187, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/handlers.py", line 201, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing

@phuocdn
Copy link

phuocdn commented Oct 1, 2021 •
edited
Loading

Openedx uses staff_graded xblock for processing and storing data to backend storage (filesystem or s3). On production ,we set up to use s3 storage but there is a bug of django-storages when save a file to s3 with a string ContentFile arg.
jschneier/django-storages#708
Because this issue have not solved yet, we have to encode the string arg before save it. We can work around by updating super-csv package (staff_graded xblock use this package to process file).
https://github.com/edx/super-csv/blob/c5a716a71dcbaca1b3dbb8ed492a76bae0fd3267/super_csv/models.py#L72
https://github.com/edx/super-csv/blob/c5a716a71dcbaca1b3dbb8ed492a76bae0fd3267/super_csv/mixins.py#L121

    def record_operation(cls, class_name_or_obj, unique_id, operation, data, original_filename='', user=None):
        """
        Save a CSVOperation
        """
        instance = cls(
            class_name=cls._get_class_name(class_name_or_obj),
            unique_id=unique_id,
            operation=operation,
            original_filename=original_filename,
            user=user,
        )
        instance.data.save(uuid.uuid4(), ContentFile(data))
        return instance

Tested on local:
Imported Scores successfully after config s3 as storage backend & encode the data above on local open edx koa release.
image
Bucket contain data: https://cloud.digitalocean.com/spaces/edxuploads?i=2b4d6e&path=csv%2F

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@phuocdn phuocdn

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@phuocdn @lamnguyen8075