Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 365142 - Linux specific API used in AIX libpty -> NULL dereference
Summary: Linux specific API used in AIX libpty -> NULL dereference
Status: RESOLVED FIXED
Alias: None
Product: CDT
Classification: Tools
Component: cdt-core (show other bugs)
Version: 7.0.2   Edit
Hardware: Power PC AIX
: P3 normal (vote)
Target Milestone: 8.0.2   Edit
Assignee: Chris Recoskie CLA
QA Contact: Doug Schaefer CLA
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-11-29 17:13 EST by Chris Recoskie CLA
Modified: 2011-11-29 18:23 EST (History)
0 users

See Also:


Attachments
patch (1.24 KB, patch)
2011-11-29 17:34 EST, Chris Recoskie CLA
recoskie: iplog-
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Chris Recoskie CLA 2011-11-29 17:13:08 EST
In openpty.c, there is the following code:

int
ptym_open(char * pts_name)
{
	int fdm;
	char *ptr;

	strcpy(pts_name, "/dev/ptmx");
	fdm = getpt();


Unfortunately, getpt() doesn't exist on AIX.  There is no compile or link error as implicit int is assumed for the declaration of getpt(), and external symbols are not resolved during a shared library link.  This means PTY support has been broken on AIX for a long, long time.  We need to use it though for RDT so we stumbled upon this problem.

The fix is a one liner... we need to call posix_openpt(O_RDWR) instead.
Comment 1 Chris Recoskie CLA 2011-11-29 17:34:05 EST
Created attachment 207695 [details]
patch
Comment 2 Chris Recoskie CLA 2011-11-29 17:53:54 EST
Committed to cdt_7_0, cdt_8_0, and master.
Comment 3 CDT Genie CLA 2011-11-29 18:23:02 EST
*** cdt git genie on behalf of Chris Recoskie ***

    Bug 365142 - Linux specific API used in AIX libpty -> NULL dereference

[*] http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=1d7cfbe63596cb90b5f9889c8f55b05de6c23c4c
Comment 4 CDT Genie CLA 2011-11-29 18:23:03 EST
*** cdt git genie on behalf of Chris Recoskie ***

    Bug 365142 - Linux specific API used in AIX libpty -> NULL dereference

[*] http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=abae07b394f4ba84c309889ee6bbeadeb2cfb20d
Comment 5 CDT Genie CLA 2011-11-29 18:23:04 EST
*** cdt git genie on behalf of Chris Recoskie ***

    Bug 365142 - Linux specific API used in AIX libpty -> NULL dereference

[*] http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=d4ea43f6c86ad260403c8af96ddfd05e2f3beaca