Issue1645 followup Fix PreferencesComponent file size validation behavior

[GoodKimchi]

Description

This pull request continues the work from PR #1603 and addresses issue #1645.

The change restores the intended client-side validation behavior for the default file size settings in PreferencesComponent.tsx. The maximum validation bound is restored to Infinity, which prevents invalid file size values from being submitted while keeping the validation consistent with similar fields on the meter and unit pages.

This pull request also removes a duplicated line in src/server/routes/preferences.js that was introduced during a merge conflict and fixes the related indentation.

Contributors:

• Oscar Bedolla (@GoodKimchi)
• Andrei Solomon Duque (@aduques)

Fixes #1645

Type of change

• Note merging this changes the database configuration.
• This change requires a documentation update

Checklist

• I have followed the OED pull request ideas
• I have removed text in ( ) from the issue request
• You acknowledge that every person contributing to this work has signed the OED Contributing License Agreement and each author is listed in the Description section.

Limitations

npm run check passes.

npm run test one failure caused by a PostgreSQL deadlock in the test:

Retrieves 15 min reading when asked for 1 hour interval so looking at hourly readings

The reported error was deadlock detected. This is a known test issue currently affecting the development branch and is unrelated to the changes in this pull request. @aduques did not have this issue

[huss]

@GoodKimchi & @aduques Thank you for this contribution. I've made two comments on specific code to consider. I did not do a full review because I am unsure about the changes. It appears you now check if values are invalid at save and pop up a toast message if they are. What OED does on other pages is, when possible/reasonable, detect bad input when it happens, put a red error box around the item, put a red error message below and disable the save button. This means the user immediately knows the issue and cannot save the bad values. This should also be correctly carried to the unsaved warnings so it also will not save. There are limited circumstances where OED does wait until save to check values but that should be where immediate checks cause issues or are too complicated/interrelated to do at field entry. I'm not sure those apply here so I'm seeking your thoughts on whether the popup on save makes sense or if it should be the other way. I'm also noting the discard button does not work but I don't know yet if that is from your changes. Once these are settled/addressed, I should be able to do a fuller review. Please let me know if anything is not clear or you have thoughts/questions.

[huss]

I'm feeling as if I am having trouble expressing my thoughts so they are clear. If you want to talk about this verbally then just let me know. Having said that, let me add this. I was not trying to propose that the error popup happens on page load/reload. What OED has on other pages (and I think here too) is that red boxes with error messages are placed on the page when issues are detected and the save is disabled. If associating with a given input box does not make sense then (I think) the message goes at the bottom of the page just before the save button. The idea was that this should make it clear why the save is disabled. (For now I'm not discussing a modification will happen on save to clean up the input that still allows the save.) If this is correct, then I was wondering why recheck on save on the preferences page since they should never get there if there are issues - at least in normal cases. If that is not the case then I would like to discuss which ones and why. Now, every page should display if the server rejects the request and why since the page checks either miss something or cannot readily check for issues. That is the final fail safe. That is okay and acceptable. Does this make more sense? I hope I'm understanding what you are saying.

Note PR #1672 changed the login/logout route to loginLogout/ and added subroutes. It was recently merged and is likely causing the merge conflict. Please let me know if you want any assistance in resolving it.

[aduques]

If associating with a given input box does not make sense then (I think) the message goes at the bottom of the page just before the save button.

I understand that having the showErrorNotification was not what you meant. I'm interpreting it that we should apply the red box / red text error messages that are used throughout OED to highlight textboxes with invalid inputs to the save button when it is disabled. The red box/red text errors will display the updated messages that specify which flags are raised in the error checks to let the user know why they can't save.

If this is correct, then I was wondering why recheck on save on the preferences page since they should never get there if there are issues - at least in normal cases. If that is not the case then I would like to discuss which ones and why.

I want to clarify that when I previously said,

"The TODO DEBUG was used to display the error message during the case of invalid inputs somehow bypassing the invalidFuncs checks. With the changes, the error message is retained, but the new addition will not affect the implementation of failed.to.submit.changes in the unsaved warning component."

I was not referring to a known bug that allows the user to bypassing the invalidFuncs check. I talked about this to describe the TODO DEBUG case where the invalidFuncs validation checks were removed to test out the updated error messages that displayed specific functions from invalidFuncs. I have not experienced any bugs with this guardrail, and as far as I am aware, invalidFuncs' only issue was the max bounds for fileSizeLimit not being Infinity (which is addressed and fixed in this PR).

I'm assuming the "recheck on save" refers to the addition of the getInvalidFieldNames() which would call invalidFuncs() again:

  onClick={() => {
					      const invalidFieldNames = getInvalidFieldNames();
					      if (invalidFieldNames) {
						      showErrorNotification(
							      translate('failed.to.submit.changes.saved') + '(' + invalidFieldNames + translate('failed.to.submit.changes.fields')
						      );
						      return;
					      }
  
					      submitPreferences(localAdminPref)
						      .unwrap()
						      .then(() => {
							      showSuccessNotification(translate('updated.preferences'));
						      })
						      .catch(err => {
							      showErrorNotification(translate('failed.to.submit.changes.saved') + err.data);
						      });
				      }}

The reason why this helper function calls invalidFuncs() again is to strictly confirm which of the functions in invalidFuncs is raised to determine how to update the error message. I understand that the first call of invalidFuncs() would prevent the recheck from even happening as the save is disabled (the original call that is a condition for disabling save). This recheck would only work with the TODO DEBUG case where the first check is removed and allowing the save button to be available.

With that in mind, I did think of alternatives, but I was hesitant to do so. This alternative solution might work with wanting to implement the red-box/red text error to the disabled save button. This alternative solution would include modifying the invalidFuncs() to update the error message (essentially what getInvalidFieldNames() does). This would prevent a recheck on invalidFuncs() as the initial check will handle disabling the save button if needed and updating the error message. I was hesitant to do this as I was not sure how changing invalidFuncs() might affect the behaviors elsewhere (if there is any beyond PreferencesComponents.tsx, but I assume not).

I think removing the "recheck" is definitely the approach since it would not be accessible in a normal use case when the first check disables the button. Something like the alternative solution I proposed above might be worth trying if you agree.

Note PR #1672 changed the login/logout route to loginLogout/ and added subroutes. It was recently merged and is likely causing the merge conflict. Please let me know if you want any assistance in resolving it.

We are currently working on merging the new changes and we will have a follow up message if we encounter any merge conflicts.

[aduques]

We have pulled and merged the recent changes for PR #1672. We didn't experience any merge conflicts, but noticed an issue with the login/logout. The login and logout still work, but there is an error message after the logout request in console. We are assuming this is not related to a merge conflict as the option to resolve a merge conflict did not pop up and the application started up without any issues.

Initially, I ran into an issue that was preventing me from logging in with either 'test' or 'test@example.com'. However, this was quickly resolved by refreshing the page and trying again (I'm assuming the changes were not live when I initially tried). I wanted to point this out as it seems other students ran into this issue based on the discord channel for codeday-sose.

[huss]

Thanks for the info on login/out. I received several reports yesterday. I posted to the OED Discord server on the developer channel about it so see that if you didn't already. It probably requires a DB change to work completely correctly.

[huss]

Thanks for the comment about the recheck. Given the red boxes & messages catch all the known issues and stops saves, I don't think OED needs to run a second set of checks on save to clarify the error message. It seems much more likely that the error message is from something else and not the checks being done in the component. Thus, the second check will not likely add any useful information. Note OED has decided not to provide the actual error from the server but say something generic as a recommenced security change. The detailed information is in the logs. So, if I understood your message and you agree with this, I think the secondary check on save can go along with all the debug code. Then I can do what is hopefully a final sweep. Please let me know any thoughts.

[aduques]

I understand that it is a design choice for OED to not include these actual error messages to the users and instead shows a generic error message. Therefore, the specific error messages will be scrapped from this PR.

However, I wanted to show you what the specific error messages would look like as a red text/ red box error on the save button. This code is removed from the PR and will not be committed, but I thought it would be interesting to consider adding these messages to OED where buttons would be disabled and it could give info to users as to why.
I've kept the specific messages as placeholder, but it could easily be replaced with generic messages to follow the design choice. I'd imagine this being something potentially helpful to users, but this idea would need to be refined more to give users information that is helpful and not already provided elsewhere. In addition, if this were to be considered in the future, it would require a better design implementation visually as I found what I've tried out to not be as polished visually.

The changes involve assigning the result of invalidFuncs to a variable, invalidFuncsResult, which allows it to be reused and only have one check as desired. We are able to get specific error messages by creating a dictionary/map which has the field name for each function in invalidFuncs. If the function returns true, the function name is used as a key to look up the field name in its bucket. The action of checking invalidFuncs and updating the error message is in the same pass and would not require a second check.

To summarize the code that will be commited in this PR, it will strictly be the requested changes of the original PR we were tasked with. The secondary check, specific error messages, and the debug code will all be removed for the final sweep.

I apologize for the time spent looking at the proposed ideas with the updated error messages for it to not be included in this PR, but I hope the basis of it can inspire future additions to OED.

[huss]

Please down worry at all about the discussion. I find your ideas interesting and they make me think. I'm open to UI changes but I want to be sure I understand the change. It can also be propose to a wider audience to see if they want it (I don't override, in general, the consensus of the group.) Currently OED tries to show each issue on the input page with a red box/message and not allowing save. Your proposal seems a similar look but done during submit. Is it correct that a user should not see your new messages if it is properly covered by the current style that stops saving? If it is only other ones and you need to know in advance to create your map then could these be added to the current checks that disable saving? If it involves an unexpected error then I'm unsure how OED could help the user with a message without showing the actual error (which we don't want to do for other potential issues). My hope is that over time the save button will not allow many errors to come back where the user sees the generic error pop up.

I still feel I'm missing something and I welcome your thoughts.

[aduques]

Your proposal seems a similar look but done during submit. Is it correct that a user should not see your new messages if it is properly covered by the current style that stops saving?

In the screenshot of the error message below the save button, it appears with the current behavior of the save button. So, the error message below the submit does not appear during the submit request, but instead it gets updated when the invalidFuncs() is called during the check. In other words, the current validation behavior that determines if the saved button is disabled is kept exactly the same. The only addition is that the error message would be displayed at the bottom to let the users know why it's still disabled.

As before, I am unaware of ways that users will be able to fail the invalidFuncs() checks (or any other unexpected errors that will raise these error messages), so I cannot provide a scenario at the moment. While I acknowledge that the error message itself can provide useful information, the current guardrails prevent the user as much as possible from even seeing these error messages (as intended).

If interested, here was the code snippets that I left out from the PR that were related to that error message:

  const fieldTranslationIds: { [key: string]: string } = {
      readingFreq: 'default.meter.reading.frequency',
      minDate: 'default.meter.minimum.date',
      maxDate: 'default.meter.maximum.date',
      readingGap: 'default.meter.reading.gap',
      meterErrors: 'default.meter.maximum.errors',
      warningFileSize: 'default.warning.file.size',
      fileSizeLimit: 'default.file.size.limit'
  };
  
  // Computed once by iterating invalidFuncs a single time - reused for the Submit button's disabled
  // state, the FormFeedback visibility, and the detailed list of which fields are invalid.
  const invalidFuncsResult = Object.entries(invalidFuncs)
      .filter(([, check]) => check())
      .map(([key]) => translate(fieldTranslationIds[key]))
      .join(', ');
  
  
  <div className='d-flex justify-content-end mt-3'>
      <Button
	      type='button'
	      onClick={discardChanges}
	      disabled={!hasChanges}
	      style={{ marginRight: '20px' }}
	      color='secondary'
      >
	      {translate('discard.changes')}
      </Button>
      <Button
	      type='submit'
	      onClick={() => {
		      submitPreferences(localAdminPref)
			      .unwrap()
			      .then(() => {
				      showSuccessNotification(translate('updated.preferences'));
			      })
			      .catch(err => {
				      showErrorNotification(translate('failed.to.submit.changes') + err.data);
			      });
	      }}
	      disabled={!hasChanges || !!invalidFuncsResult}
	      color='primary'
      >
	      {translate('submit')}
      </Button>
  </div>
  <FormFeedback style={{ display: invalidFuncsResult ? 'block' : 'none' }}>
      {translate('failed.to.submit.changes.saved') + '(' + invalidFuncsResult + translate('failed.to.submit.changes.fields')}
  </FormFeedback>

[huss]

Thank you for explaining this several times. I think I am finally starting to understand. I think the idea is fine but I'm not yet convinced it is worth doing across the pages. Here is my current thinking:

• It is code that needs to be maintained.
• The pages we are discussing are all admin-only pages. This means they are not the typical OED user and they are much more likely to look at the help pages and/or have experience with the UI. If these were used by many users then that would be different. For example, OED works to remove choices from graphics pages that are not allow and shows a message in large font on the center of the graphic or as popups that are specific.
• Many admin pages would not need scrolling to see the red box and text so it isn't that hard to see if you know to look for it (whenever the save is disabled).
• This is a common web UI technique. I acknowledge some fancy pages jump to the bad value(s) on save but that would be more work for what (I perceive) as small payback on admin pages.

As I said before, you are making me think about alternatives. Given what you have said and done, this already come to mind. A non-specific message could be placed near the save button to emphasize that the admin needs to find all the red boxes and fix them. This might help because OED does use red boxes for items that will be modified on save from what is shown and this clearly shows the difference. The unclickable save button does that too but if people think this would help then I'm open/okay with this. This would be consistent across pages so maintaining it would be much easier. I'm unsure if it would be easy to do but I would think it isn't too bad.

Let me know what you think and you can still advocate for your original solution.

[aduques]

I agree that my proposed code would primarily be used by admin users who will interact with the create/edit features, whereas actual non-admin users would never see these changes. I wanted to suggest these changes, but I can understand that at the moment it has niche applications.

I support the idea of having a generic message be displayed throughout the pages that explain why the buttons are disabled, but I understand that the effectiveness varies on each page. I believe that it would be important to make sure that the generic message doesn't just repeat what the red text/red box error messages display to be effective.

I think that my original solution is not ready for implementation and it has clear room to be improved on. This idea should definitely be reconsidered if OED has gotten a lot of reports from users stating they aren't clear as to why they cannot access the save buttons. However, the existing red text messages should already clear any confusion, so it is likely not the case. For right now, I think it would make sense to put this idea on hold until it becomes apparent that users need changes to clear any future confusions that they might have. Or, it would also make sense to revisit this idea if there is a future request to improve the UI to make it more new user friendly in any way.

As I continue to work on OED, I will keep a look out for any features that might be potentially confusing for end users and discuss these with you when they arise.

[huss]

As I continue to work on OED, I will keep a look out for any features that might be potentially confusing for end users and discuss these with you when they arise.

I appreciate your thoughts so far and any you have in the future. OED welcomes them.

[huss]

@GoodKimchi & @aduques Thank you for the updated code and your willingness to reconsider items done. I have now reviewed and tested. I made one comment to consider. I'm open to ideas on what to do and when/who might address it. Please let me know any thoughts.

[huss]

I played with this and it isn't showing the max warning as I had hoped. I debugged for a little while but not completely. It seems to relate to the fact that it is using a general function (makeLocalChanges) to handle changes and not one that thinks it is a number. This may unwind back to how it is stored in OED (here and/or server). I looked at src/client/app/components/meters/CreateMeterModalComponent.tsx and it does it differently but seems to check the max value.

Another issue is that src/server/routes/preferences.js has a max of 1000000000. I'm unsure where that came from. src/server/routes/meters.js on maxVal does not seem to have any limit and assumes it could not come through as a number if it was invalid (I think). I'm unclear on why OED has an upper limit on preferences and if the one in meter does what is desired. Again, it may relate to treating as a string instead of a number.

I could not add above so am putting here. The date check here is close to but a little different than meter. Unsure if it matters but wonder about standardizing them.

The old PR has a comment that also included unit for some of these. That may also help/need coordination.

I'm unsure the right overall solution for this and welcome thoughts.

[GoodKimchi]

Hi @huss, @aduques and I dug into this a bit more and reproduced the issue. When entering a very large value, such as 9999999999999, for Default File Size Limit, the frontend accepts it without inline validation, but the backend rejects it with a Bad Request because preferences.js currently validates both file-size fields with a maximum of 1000000000.

I noticed you mentioned you were unsure where that limit came from, so I was thinking about a few possible directions:

1. Keep the current maximum (1000000000) and align the frontend validation/messages with the backend.
2. Keep a finite maximum, but define it as a shared/documented constant so the frontend and backend can't drift apart in the future.
3. Remove the application-level maximum and allow any non-negative value if the intent is for deployments to determine their own upload limits.

Since these settings appear to be used mainly for CSV or spreadsheet imports, I would not expect most deployments to approach a 1 GB upload. That made me wonder whether the current limit is an intentional design decision or primarily a safeguard against unrealistic values. A shared constant seems maintainable and would be easy to update later if larger files need to be supported. I’m happy to implement whichever direction makes the most sense.

While tracing this, I also noticed what appears to be the same type of frontend/backend validation mismatch for Default Meter Reading Gap: the frontend effectively allows values above the backend maximum. I think that should probably be handled in a separate issue or PR to keep this one focused, but I wanted to mention it because it appears to be the same class of bug.

[huss]

@GoodKimchi & @aduques Thank you for thinking this through. My thoughts are below.

dug into this a bit more and reproduced the issue. When entering a very large value, such as 9999999999999, for Default File Size Limit, the frontend accepts it without inline validation, but the backend rejects it with a Bad Request because preferences.js currently validates both file-size fields with a maximum of 1000000000.

I noticed you mentioned you were unsure where that limit came from, so I was thinking about a few possible directions:

1. Keep the current maximum (1000000000) and align the frontend validation/messages with the backend.

2. Keep a finite maximum, but define it as a shared/documented constant so the frontend and backend can't drift apart in the future.

3. Remove the application-level maximum and allow any non-negative value if the intent is for deployments to determine their own upload limits.

Since these settings appear to be used mainly for CSV or spreadsheet imports, I would not expect most deployments to approach a 1 GB upload. That made me wonder whether the current limit is an intentional design decision or primarily a safeguard against unrealistic values. A shared constant seems maintainable and would be easy to update later if larger files need to be supported. I’m happy to implement whichever direction makes the most sense.

Yes. OED should not allow files that are too large. 1 GB seems more than enough. Aligning the front/back-end is a good idea and centralizing so they can never differ is even better. You can proceed with this unless you have thoughts/questions.

While tracing this, I also noticed what appears to be the same type of frontend/backend validation mismatch for Default Meter Reading Gap: the frontend effectively allows values above the backend maximum. I think that should probably be handled in a separate issue or PR to keep this one focused, but I wanted to mention it because it appears to be the same class of bug.

Nice catch. I agree they should be consistent. Reading gap is in seconds so 86400 is 60 days. That was undoubtedly deliberate as a more than max value to ever use. I think it would stop really silly values while still allowing it to be so big that one can turn off the warnings about gaps. So, that value should consistently be used. Do you want to create the issue or should I? If you want to work on it then put in a comment on the new issue or let me know.

I welcome any thoughts.

[GoodKimchi]

Thanks @huss . That all makes sense. @aduques and I will go ahead with the shared constant approach for the file-size validation so the frontend and backend stay in sync.
I'd also be happy to work on the Reading Gap validation afterward. I'll create a separate issue to keep this PR focused, then submit a follow-up PR for that consistency fix as well.

[GoodKimchi]

Thanks for the guidance, @huss . We implemented the shared constant approach and pushed the changes.

The frontend and backend now both use a shared MAX_FILE_SIZE_LIMIT constant from src/common, so the validation is consistent and the frontend prevents values that the backend would reject.

We kept the Reading Gap changes out of this PR as discussed to keep it focused. @aduques has reviewed and tested the changes, so I believe it's ready for your review. I'll open a separate issue and follow-up PR for the Reading Gap consistency changes.

[huss]

@GoodKimchi & @aduques Thank you for the updated code. Overall, it is good. I have made two comments to consider. I want to note that I think all comments before this one are now covered by newer comments or resolved. Please let me know if anything is not clear or you have thoughts/questions.

[huss]

First, this new message is off since it indicated zero is not allowed. If that is why you added it then it can go. Second, I don't think it is needed. It isn't done on other pages. I tried removing the conditional and true part and I get all the messages desired. Thus, if not for the first item I still think this should go. Do you agree?

[aduques]

I agree that since a value less than 0 cannot be typed (negative value), then there is no case that the "error.greater" message is displayed. Therefore, that condition is never reached. When the condition is removed, then the desired messages are still displayed. I have updated the FormFeedback line to the following:

  <FormFeedback>
				      <FormattedMessage
					      id="error.bounds"
					      values={{
						      min: 0,
						      max: Number(localAdminPref.defaultFileSizeLimit)
					      }}
				      />
  </FormFeedback>

[huss]

I'm uncertain when the min is needed. It is not in the form feedback below. Since the defaultFileSizeLimit cannot exceed MAX_FILE_SIZE_LIMIT I thought that the check on defaultFileSizeLimit would be sufficient. What do you think?

[aduques]

I agree that min in this case is redundant. I have removed the min property and have confirmed that you cannot type a value below 0 regardless. I believe that this min property was initially included to capture a case where the user might attempt to input a minimum value. I have not investigated what exactly prevents no negative values from being inputed, but I can confirm that the min property is not responsible and can be removed.

[huss]

Accidentally approved so now marking as Requested changes.

[huss]

We kept the Reading Gap changes out of this PR as discussed to keep it focused.

I'm trying to get this to where only new comments apply so I want to check if you plan to open an issue and/or work on this? If not, I can do the issue.

[huss]

What OED does on other pages is, when possible/reasonable, detect bad input when it happens, put a red error box around the item, put a red error message below and disable the save button. This means the user immediately knows the issue and cannot save the bad values. This should also be correctly carried to the unsaved warnings so it also will not save.

Sorry I missed this before but am now sweeping all previous comments. When I set the warning size to too big a value, it shows a red box and disables the save button. However, if I click on the OED icon (without discarding so I try to leave the page) then the unsaved warning popup still has the save enabled. I was able to save an invalid value to the DB.

I looked into why. First, the route cannot easily check this because it depend on another routed value. However, I think some routes do checks outside the param validation to allow this type of check. The DB does not stop this (maybe it should). Second, src/client/app/components/SimpleUnsavedWarningComponent.tsx has a prop that disables the save button but src/client/app/components/UnsavedWarningComponent.tsx does not. That would need to be fixed to get the save button to be disabled.

Do you have thoughts on this? If you agree, I think it would be fine to open an issue (I can do it) to resolve these as they are not a result of your changes.

[aduques]

Sorry I missed this before but am now sweeping all previous comments. When I set the warning size to too big a value, it shows a red box and disables the save button. However, if I click on the OED icon (without discarding so I try to leave the page) then the unsaved warning popup still has the save enabled. I was able to save an invalid value to the DB.

I looked into why. First, the route cannot easily check this because it depend on another routed value. However, I think some routes do checks outside the param validation to allow this type of check. The DB does not stop this (maybe it should). Second, src/client/app/components/SimpleUnsavedWarningComponent.tsx has a prop that disables the save button but src/client/app/components/UnsavedWarningComponent.tsx does not. That would need to be fixed to get the save button to be disabled.

Do you have thoughts on this? If you agree, I think it would be fine to open an issue (I can do it) to resolve these as they are not a result of your changes.

Thanks for the review. We agree with both points. We'll remove the extra conditional FormFeedback logic and simplify the max prop since the defaultFileSizeLimit validation already guarantees it cannot exceed MAX_FILE_SIZE_LIMIT.
We also agree that the UnsavedWarningComponent behavior appears to be a separate issue from this PR. We'd be happy to work on that as a follow-up issue, and we can create the issue soon.

I want to give an insight on the differences between UnsavedWarningComponent.tsx and SimpleUnsavedWarningComponent.tsx as someone who has created the SimpleUnsavedWarningComponent.tsx. SimpleUnsavedWarningComponent.tsx was created as UnsavedWarningComponent.tsx has been hardcoded to work strictly with PreferencesComponents.tsx (Admin Settings page) and could not cleanly because the handleSubmit logic was coded directly into the component. The reason is that the handleSubmit logic not be universally used with the different schemas on different routes. The handleSubmit could only work with the requests and endpoints related to PreferencesComponents.tsx.

The creation of SimpleUnsavedWarningComponent.tsx separates the handleSubmit logic so that it is compatible with the existing submit logic on the other client files. I'm assuming that issues after mine related to updating and debugging files that used SimpleUnsavedWarningComponent.tsx have overlooked UnsavedWarningComponent.tsx usage in PreferencesComponent.tsx. In other words, I should have made sure that UnsavedWarningComponent.tsx had the same functionality and behavior as SimpleUnsavedWarningComponent.tsx in that initial issue.

[huss]

I saw the update. I'm sorry that it is going to take a little while for me to get to this. Let me know if the delay becomes an issue for you.