Tap coordinates are incorrect after device rotation #5

Activity

1. pg8wood

   added a commit that references this issue on Oct 8, 2025

   Force portrait mode until axe can handle rotation

   ...

   de4c729
2. paul-foreflight commented on Nov 5, 2025

   

   paul-foreflight

   on Nov 5, 2025

   Last edited by paul-foreflight

   More actions

   would definitely have preferred to figure out what the issue is here on the util side, but 90% of my experience is doing stuff in python.

   so its not exactly trivial, but I was able to make a workaround. it assumes that you have the ability to parse elements from axe's describe ui functionality and an element that will always be on screen

   the first method is a function of my AXeElement's class (you'll have to make your own or figure something else out), and this orientation correction will normally be applied when they are created (we're going to do a special use case of course to find the current orientation below), if you want, you could always stop here and force whichever orientation you want.

   If you're like me and it just itches at your brain that in your CI/CD env. or when other people run the tests you don't want to say "it has to be in the orientation that you set in the .env file" then you'll want to look at the other methods.

   You'll want to find an element that is always on screen, and you can select, or something that causes another event to happen on screen (lock and unlock are the same button, but change names when clicked)

   create your equivalent of find_orientation script, try each possible position you can be in, find it, click it, validate click did something (or not) and return the orientation. Now we know our orientation. Yay.

   Also for what its worth I did look into other ways of checking for the sim's orientation, and came up with nothing besides installing a separate app? I dunno maybe I'll ask my app side people if there's some way they can report that from the app itself, shove it in some element or other

     def _apply_orientation_correction(self, orientation):
           # Get the original frame data (assumed to be from PORTRAIT mode)
           self.frame = self.element_data.get('frame', {})
           x_orig = self.frame.get('x')
           y_orig = self.frame.get('y')
           w_orig = self.frame.get('width')
           h_orig = self.frame.get('height')
   
           screen_info = AXeUtils.get_screen_info(self.udid, logger=self.logger)
           p_width = screen_info.get('width')  # Portrait Width (shorter side)
           p_height = screen_info.get('height') # Portrait Height (longer side)
   
           # Start with new values defaulting to original values
           x_new = x_orig
           y_new = y_orig
           w_new = w_orig
           h_new = h_orig
   
           try:
               if orientation == 'LANDSCAPE':
                   # 90° clockwise (home button right)
                   # The new top-left corner is the old top-right corner, rotated.
                   x_new = y_orig
                   y_new = p_width - (x_orig + w_orig)
                   w_new = h_orig
                   h_new = w_orig
   
               elif orientation == 'PORTRAIT_UPSIDE_DOWN':
                   # 180° rotation
                   # The new top-left corner is the old bottom-right corner.
                   x_new = p_width - (x_orig + w_orig)
                   y_new = p_height - (y_orig + h_orig)
                   # w_new and h_new are unchanged
                   
               elif orientation == 'LANDSCAPE_FLIPPED':
                   # 270° clockwise / 90° counter-clockwise (home button left)
                   # The new top-left corner is the old bottom-left corner, rotated.
                   x_new = p_height - (y_orig + h_orig)
                   y_new = x_orig
                   w_new = h_orig
                   h_new = w_orig
               
               # else: orientation is 'PORTRAIT', no changes needed
           # Apply the new, corrected frame values
               self.frame['x'] = x_new
               self.frame['y'] = y_new
               self.frame['width'] = w_new
               self.frame['height'] = h_new
   
           except Exception as e:
               if self.logger:
                   self.logger.debug(f"Orientation correction failed, using original coordinates: {e}")
               # Ensure frame is reset to original values if any math fails
               self.frame['x'] = x_orig
               self.frame['y'] = y_orig
               self.frame['width'] = w_orig
               self.frame['height'] = h_orig
   
   
       def get_screen_info(self) -> str:
           """Determine device orientation by checking lock/unlock button visibility."""
           screen_info = self.get_screen_dimensions()
           orientation = 'UNKNOWN'
           self.logger.info(f"Screen info for orientation check: {screen_info}")
           orientations_to_check = ['PORTRAIT', 'LANDSCAPE', 'PORTRAIT_UPSIDE_DOWN', 'LANDSCAPE_FLIPPED']
           
           for orientation in orientations_to_check:
               screen_info['orientation'] = orientation
               element_data = axe_utils.find_element_in_ui(self.udid, lock_button, logger=self.logger)
               lock_element = AXeElement(self.udid, element_data, screen_info, self.logger)
               lock_element.tap()
               self.logger.info(f'{orientation}: {lock_element.frame}')
               try:
                   if unlock_element_data := axe_utils.find_element_in_ui(self.udid, unlock_button, logger=self.logger):
                       unlock_element = AXeElement(self.udid, unlock_element_data, screen_info, self.logger)
                       unlock_element.tap()
                       return screen_info
               except Exception:
                   pass
       
       @staticmethod
       def get_screen_info(udid: str, logger=None) -> Dict[str, int]:
           """Get screen information using AXe CLI."""
           app_element = AXeUtils.find_element_in_ui(udid, AXeUtils.application, logger=logger)
           if app_element and 'frame' in app_element:
               return app_element['frame']
           else:
               if logger:
                   logger.warning("Could not find application element or frame data, using default screen info")
               # Return default iPad Pro dimensions if we can't get actual screen info
               return {'x': 0, 'y': 0, 'width': 834, 'height': 1194}
   
3. cameroncooke

   added a commit that references this issue on May 10, 2026

   fix(orientation): Auto-detect simulator coordinate rotation

   ...

   This issue will close once commit abaa9ae is merged into the 'main' branch.

   abaa9ae
4. cameroncooke

   closed this as completedin #44on May 10, 2026
5. cameroncooke

   added 2 commits that reference this issue on May 10, 2026

   fix(commands): translate logical coords to physical for non-portrait …

   ...

   41a44c8

   fix(orientation): Auto-detect simulator coordinate rotation

   ...

   4fa25bf