Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 365142

Summary: Linux specific API used in AIX libpty -> NULL dereference
Product: [Tools] CDT Reporter: Chris Recoskie <recoskie>
Component: cdt-coreAssignee: Chris Recoskie <recoskie>
Status: RESOLVED FIXED QA Contact: Doug Schaefer <cdtdoug>
Severity: normal    
Priority: P3    
Version: 7.0.2   
Target Milestone: 8.0.2   
Hardware: Power PC   
OS: AIX   
Whiteboard:
Attachments:
Description Flags
patch recoskie: iplog-

Description Chris Recoskie CLA 2011-11-29 17:13:08 EST
In openpty.c, there is the following code:

int
ptym_open(char * pts_name)
{
	int fdm;
	char *ptr;

	strcpy(pts_name, "/dev/ptmx");
	fdm = getpt();


Unfortunately, getpt() doesn't exist on AIX.  There is no compile or link error as implicit int is assumed for the declaration of getpt(), and external symbols are not resolved during a shared library link.  This means PTY support has been broken on AIX for a long, long time.  We need to use it though for RDT so we stumbled upon this problem.

The fix is a one liner... we need to call posix_openpt(O_RDWR) instead.
Comment 1 Chris Recoskie CLA 2011-11-29 17:34:05 EST
Created attachment 207695 [details]
patch
Comment 2 Chris Recoskie CLA 2011-11-29 17:53:54 EST
Committed to cdt_7_0, cdt_8_0, and master.
Comment 3 CDT Genie CLA 2011-11-29 18:23:02 EST
*** cdt git genie on behalf of Chris Recoskie ***

    Bug 365142 - Linux specific API used in AIX libpty -&gt; NULL dereference

[*] http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=1d7cfbe63596cb90b5f9889c8f55b05de6c23c4c
Comment 4 CDT Genie CLA 2011-11-29 18:23:03 EST
*** cdt git genie on behalf of Chris Recoskie ***

    Bug 365142 - Linux specific API used in AIX libpty -&gt; NULL dereference

[*] http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=abae07b394f4ba84c309889ee6bbeadeb2cfb20d
Comment 5 CDT Genie CLA 2011-11-29 18:23:04 EST
*** cdt git genie on behalf of Chris Recoskie ***

    Bug 365142 - Linux specific API used in AIX libpty -&gt; NULL dereference

[*] http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=d4ea43f6c86ad260403c8af96ddfd05e2f3beaca