Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uia in ms word by default #8919

Merged
merged 11 commits into from Jan 15, 2019
Merged

Uia in ms word by default #8919

merged 11 commits into from Jan 15, 2019

Conversation

michaelDCurran
Copy link
Member

@michaelDCurran michaelDCurran commented Nov 5, 2018

Link to issue number:

Closes #8502
Closes #7409

Summary of the issue:

Microsoft Word 2016 exposes a rich UI Automation implementation. For some time now, users have been able to optionally turn this on with a hidden config value. It seems that NVDA's support for MS Word via UIA has major performance advantages over the older object model support, so NVDA should use the UIA support by default where available. However, as the UIA implementation improved throughout Office 2016's lifetime, we should only enable our support for recent builds of Office 2016.

Description of how this pull request fixes the issue:

This PR causes NVDA to support MS Word via UI Automation for Office 2016 build 9000 and higher. Users on lower builds can still force UIA support with the hidden config value.
This PR also ensures that the MS Office ribbon is accessed with UI Automation, both when expanded and collapsed, which solves some issues with focus not reporting for some ribbon controls.
As a part of this change, it was necessary to either move, or come up with an alternative fix, for a couple of previously solved issues. Namely supporting the autoComplete list in Outlook (#2816), and the comboboxes in the MS Office Options dialogs (#4056), as they use the NetUI framework just like the Ribbon.

Testing performed:

  • Opened a Word document in MS Word 2016 build 11029, config value UIA.useInMSWordWhenAvailable was false. Confirmed that NVDA used UI Automation to access the document.
  • In both Outlook 2010 and 2016 build 11029, created a new message, and started typing a known contact into the To: field. NVDA automatically reported the first autoComplete contact.
  • Opendd the MS Office Options dialog for MS Word 2010 and 2016 build 11029 and tabbed to the first combo box. The name was correctly reported for the control.
  • In MS Word 2016 build 11029, Moved to the Ribbon with alt, arrowed to the Insert tab. As ribbon was collapsed, pressed enter. Tabbed through options in Insert ribbon. All focus changes were reported, and all controls were accessed using UI Automation.

Known issues with pull request:

None.

Change log entry:

Bug fixes:

  • Significant speed improvements when navigating documents in Microsoft Word and Outlook 2016 build 9000 or higher, thanks to NVDA's UI Automation support for Microsoft Word now used by default for these builds.
  • NVDA no longer fails to report the focus for some controls in the Microsoft Office 2016 ribbon when collapsed.
  • NVDA no longer fails to report the suggested contact when entering addresses in new messages in Outlook 2016.

…roblems directly: Office 2010 options dialog comboboxes have no name, and Outlook pre 2016 autoComplete list should use MSAA.
@LeonarddeR
Copy link
Collaborator

I haven't yet looked at the code.

I recall that there is at least still one outstanding issue with UIA. Table navigation is currently a bit broken, empty cells are considered out of table.

Does this implementation also allow you to switch back to the object model for Office 2016 build 9000 and above?

@michaelDCurran
Copy link
Member Author

michaelDCurran commented Nov 5, 2018 via email

@LeonarddeR
Copy link
Collaborator

@bramd: Could you think of any issues that need to be addressed beforehand?

@lukaszgo1
Copy link
Contributor

Was this tested on Windows 7? I recall reading about freezes when Office 2016 was used with UIA enabled on Win 7, but i have no way to test this easily at the moment.

@michaelDCurran
Copy link
Member Author

michaelDCurran commented Nov 5, 2018 via email

and not config.conf['UIA']['useInMSWordWhenAvailable']
):
# We can only safely check the version of known Office apps using the Word document control.
# Other uses for now we just need to assume the implementation is good.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is not clear to me yet.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this comment could be reworded. Are you saying that "for unknown office apps, which we can not safely check the version using the Word document control, we must just assume the implementation is good"?
Are we assuming that the UIA implementation is good?

try:
versionMajor,versionMinor,versionBuild,versionPatch=[int(x) for x in appModule.productVersion.split('.')]
except Exception as e:
log.error("Error parsing versioninformation %s, %s"%(appModule.productVersion,e))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing space between version and information

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This space is still missing.

@@ -156,9 +164,15 @@ def event_NVDAObject_init(self,obj):
obj.role=controlTypes.ROLE_LISTITEM

def chooseNVDAObjectOverlayClasses(self, obj, clsList):
# Currently all our custom classes are IAccessible
if isinstance(obj,UIA) and obj.UIAElement.cachedClassName in ("LeafRow","ThreadItem","ThreadHeader"):
clsList.insert(0,UIAGridRow)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you might one to return in this block, otherwise the logic under it is needlessly executed (i.e. fetching the role)

@enessaribas
Copy link

hi,
Are there plans to implement this for powerpoint and excel as well?

@dnz3d4c
Copy link

dnz3d4c commented Nov 7, 2018

An error occurs when you activate UIA, enable Browse mode, and then press H, the key to go head to head. Here is a log of the error:
ERROR - scriptHandler.executeScript (10:16:12.188):
error executing script: <bound method WordBrowseModeDocument.script_nextHeading of <NVDAObjects.UIA.wordDocument.WordBrowseModeDocument object at 0x0B125950>> with gesture u'h'
Traceback (most recent call last):
File "scriptHandler.pyc", line 187, in executeScript
File "browseMode.pyc", line 414, in
File "browseMode.pyc", line 386, in _quickNavScript
File "UIABrowseMode.pyc", line 158, in UIAHeadingQuicknavIterator
File "UIAUtils.pyc", line 97, in getUIATextAttributeValueFromRange
UIAMixedAttributeError

  • OS: Windows 10 version 1803 / Language set: Korean

@michaelDCurran
Copy link
Member Author

michaelDCurran commented Nov 7, 2018 via email

@dnz3d4c
Copy link

dnz3d4c commented Nov 7, 2018

It is an internal document and can not be shared. However, when time permits, we will create a reproducible document.

@dnz3d4c: I need to know what exact version of Microsoft Word this was. And I will also require a sample document where the issue occurs. I cannot reproduce this in documents I have.

… property. It is no longer needed in recent versions of MS Word, and in fact it was causing tables to not be reported when positioned on a blank cell.
@michaelDCurran
Copy link
Member Author

michaelDCurran commented Nov 8, 2018 via email

@dnz3d4c
Copy link

dnz3d4c commented Nov 8, 2018

You can reproduce the results with the attached document:

  • O365 word version: Microsoft® Word MSO (16.0.11001.20070) 64bit
    test.docx

An error occurs when you activate UIA, enable Browse mode, and then press H, the key to go head to head. Here is a log of the error:
ERROR - scriptHandler.executeScript (10:16:12.188):
error executing script: <bound method WordBrowseModeDocument.script_nextHeading of <NVDAObjects.UIA.wordDocument.WordBrowseModeDocument object at 0x0B125950>> with gesture u'h'
Traceback (most recent call last):
File "scriptHandler.pyc", line 187, in executeScript
File "browseMode.pyc", line 414, in
File "browseMode.pyc", line 386, in _quickNavScript
File "UIABrowseMode.pyc", line 158, in UIAHeadingQuicknavIterator
File "UIAUtils.pyc", line 97, in getUIATextAttributeValueFromRange
UIAMixedAttributeError

* OS: Windows 10 version 1803 / Language set: Korean

@dnz3d4c
Copy link

dnz3d4c commented Nov 8, 2018

If you open the attached document, activate browse mode, and then perform the Control + End gesture twice, you will get the following error:

ERROR - scriptHandler.executeScript (12:29:10.523):
error executing script: <bound method WordBrowseModeDocument.script_bottomOfDocument of <NVDAObjects.UIA.wordDocument.WordBrowseModeDocument object at 0x0B432E90>> with gesture u'control+end'
Traceback (most recent call last):
File "scriptHandler.pyc", line 187, in executeScript
File "cursorManager.pyc", line 242, in script_bottomOfDocument
File "cursorManager.pyc", line 143, in _caretMovementScriptHelper
File "browseMode.pyc", line 1226, in _set_selection
File "baseObject.pyc", line 21, in get
File "browseMode.pyc", line 1099, in _get_focusableNVDAObjectAtStart
File "UIABrowseMode.pyc", line 189, in UIAControlQuicknavIterator
File "baseObject.pyc", line 21, in get
File "UIABrowseMode.pyc", line 345, in get_UIAElementAtStart
File "baseObject.pyc", line 21, in get
File "NVDAObjects\UIA_init
.pyc", line 345, in _get_UIAElementAtStart
File "UIAUtils.pyc", line 171, in getChildrenWithCacheFromUIATextRange
COMError: (-2146233079, None, (None, None, None, 0, None))

  • O365 word version: Microsoft® Word MSO (16.0.11001.20070) 64bit Win 10 version 1803 language set: Korean
  • Attach file: comError.docx

@michaelDCurran
Copy link
Member Author

michaelDCurran commented Nov 8, 2018 via email

@dnz3d4c
Copy link

dnz3d4c commented Nov 8, 2018

Both issues seem to have been fixed. We'll let you know if we have other issues. Thank you.

@dnz3d4c: I fixed the control+end problem. Thanks. I can also no longer reproduce the heading problem in the latest UIAInMSWordByDefault branch commits, though I could reproduce it in NVDA alpha builds. I'm assuming it may have got fixed either with the table navigation fix or the COMError fix. Here is the latest try build of this branch: https://ci.appveyor.com/api/buildjobs/a3wegd33abnyp915/artifacts/output%2Fnvda_snapshot_try-UIAInMSWordByDefault-16234%2Ce809aca6.exe

@LeonarddeR
Copy link
Collaborator

However, we may need to increase the build number from 9000 to 11000.

Yeah, this really needs some testing, also on Windows 8 if possible.

Having said this though, if there is a big push for the option from people then I'll accept it.

I myself don't require it, but the object model is required to support chards in Word.

@bramd
Copy link
Contributor

bramd commented Nov 9, 2018 via email

@josephsl
Copy link
Collaborator

Hi,

If you have time, can you take a look at #8502 comments and see if this PR can lend support for auto-complete suggestions in Office 2016/2019? Thanks.

@michaelDCurran
Copy link
Member Author

@josephsl This pr already addresses #8502, at least, suggestions are read when pressing up and down arrow. The suggestion open/closed sound is not played though.

@josephsl
Copy link
Collaborator

josephsl commented Nov 23, 2018 via email

and not config.conf['UIA']['useInMSWordWhenAvailable']
):
# We can only safely check the version of known Office apps using the Word document control.
# Other uses for now we just need to assume the implementation is good.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this comment could be reworded. Are you saying that "for unknown office apps, which we can not safely check the version using the Word document control, we must just assume the implementation is good"?
Are we assuming that the UIA implementation is good?

try:
versionMajor,versionMinor,versionBuild,versionPatch=[int(x) for x in appModule.productVersion.split('.')]
except Exception as e:
log.error("Error parsing versioninformation %s, %s"%(appModule.productVersion,e))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This space is still missing.

return True
if (
versionMajor<16
or versionMajor==16 and versionMinor==0 and versionBuild<9000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I would prefer to see these values (16, 0, 9000) put in a named constant.

if winUser.getClassName(hwnd) in ("WeekViewWnd","DayViewWnd"):
windowClass=winUser.getClassName(hwnd)
# #2816: Outlook versions before 2016 auto complete does not fire enough UIA events, IAccessible is better.
if windowClass=="NetUIHWND":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few usages of this "NetUIHWnd" can you put them in a named constant, perhaps outlookWindowClass

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe this would improve things. A window class string is as gooder constant from my point of view. Something like OutlookWindowClass does not addiquitly describe the string as this NetUIHWND class is used in more than just Outlook. It seems to be some kind of generic window class for Ribbons. But to be clear, not the only one though, there is also NetUIToolWindow. Personally I think the strings are good enough. However, I guess we could convert them in to constants I.e. NetUIHWND="NetUIHWND" but I really don't see the point.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. My hope was that we could describe the string better. As someone unfamiliar with the value, it doesn't tell you any of the things you just mentioned. Having a constant with comment you just wrote would be helpful in my opinion. Ideally a name that describes it well, it seems like that is difficult.

Copy link
Contributor

@feerrenrut feerrenrut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actions look fine, much clearer. I had one more comment to make a case for a constant for that window class value, but I'll leave the decision up to you.

@LeonarddeR
Copy link
Collaborator

I tend to agree with Mick on not seeing the necessity of having a constant here.

@Neurrone
Copy link

Since I don't use Word much myself, I had a friend try it. Do you have to do any configuration beyond downloading the snapshot and choosing "continue running"?

In a 200 page document, he didn't experience any noticeable improvement when the file was being edited with word 16.0.11209 (i.e, the usual multiple second lag between key presses).

@@ -65,6 +66,10 @@
"FoxitDocWnd",
]

Version=namedtuple('Version',('major','minor','build'))
# The minimum version of Microsoft Word where we can trust that UI Automation is complete enough to use
minMSWordUIAVersion=Version(16,0,90000)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This curently fails. I believe you wanted to say 9000 here.

@LeonarddeR
Copy link
Collaborator

This pull request does not yet exclude Windows 7. I've been able to test on one Windows 7 machine, which caused freezes. I"m not sure what build it was, though.

Might it be possible to track down the minimum required Windows version based on the version of the UIA interfaces shipped with it?

@Novalis7747
Copy link

Novalis7747 commented Jan 17, 2019 via email

@zstanecic
Copy link
Contributor

zstanecic commented Jan 17, 2019 via email

@Novalis7747
Copy link

Novalis7747 commented Jan 17, 2019 via email

@dkager
Copy link
Collaborator

dkager commented Jan 18, 2019

The UIA implementation still seems to have issues with tables in HTML e-mails in Outlook. I encountered two e-mails so far that wouldn't read at all using navigation keys on a braille display and read unreliably using the arrow keys. Downgrading to 2018.4.1 fixed the problems for both e-mails. I'll investigate more soon.

@michaelDCurran
Copy link
Member Author

I'm guessing that most people here who are having trouble with Outlook html emails are finding that browse mode is not automatically enabled when reading emails? that is what I am seeing on my system.
Hopefully this shouldn't be too hard to address.
That would explain why people are missing a lot of content in tables, as the arrow keys by default only move down the current column.
I am not seeing the traceback reported though.

@michaelDCurran
Copy link
Member Author

I am now thinking that this is still not ready for 2019.1. I want to keep all the NetUI / Outlook stuff, plus any improvements to UIA in MS Word. But, having it on by default should not happen until more things are fixed in Microsoft word itself. E.g. color issues. Layout tables in Outlook etc.
Therefore for 2019.1 we should go back to relying on the hidden useInMSWordWhenAvailable UIA config option, though perhaps consider adding the Advanced tab to the settings dialog and exposing it there, so more people experiment with the feature.
At the same time, I want to do some more performance profiling of the existing object model supprt for MS Word, and see if we can tighten things up some more.

@Novalis7747
Copy link

Novalis7747 commented Jan 23, 2019 via email

@LeonarddeR
Copy link
Collaborator

@michaelDCurran wrote:

I am now thinking that this is still not ready for 2019.1. I want to keep all the NetUI / Outlook stuff, plus any improvements to UIA in MS Word. But, having it on by default should not happen until more things are fixed in Microsoft word itself. E.g. color issues. Layout tables in Outlook etc.

I tend to agree.

... perhaps consider adding the Advanced tab to the settings dialog and exposing it there, so more people experiment with the feature.

This makes sense. It is related to #6788, though that proposal goes a bit further than a simple advanced tab.

At the same time, I want to do some more performance profiling of the existing object model supprt for MS Word, and see if we can tighten things up some more.

That would really be helpful, as there many people still relying on older version of Office, particularly people who are at Office 2016 volume licenses, or Office 2013 and older.

@codeofdusk
Copy link
Contributor

Are there plans to reconsider UIA in Word by default for a future Office/NVDA version? Have the previously-reported issues been resolved?

@Neurrone
Copy link

I remember the last time I tried it table navigation still didn't work right for some tables, along with the other issue of performance degrading after 10 minutes or so, requiring a restart of NVDA to continue working for the next 10 minutes.

@LeonarddeR
Copy link
Collaborator

LeonarddeR commented Mar 25, 2020 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet